Cloudflare Introduces Granular AI Bot Controls for Website Owners

Cloudflare has announced a significant update to its bot management features, providing website owners with granular controls over how AI bots interact with their sites. This new system moves beyond the previous single-click blanket block for all AI crawlers, allowing for more precise management of search, training, and agent bots.

The update, available to all Cloudflare customers, including those on the free plan, categorizes AI bot activity into three distinct areas: Search, Training, and Agent. This differentiation enables site administrators to set specific rules for each type of bot, optimizing site performance and content exposure according to their objectives.

Key Facts

Feature	Description
Granular Controls	Replaces blanket AI bot blocking with specific rules for Search, Training, Agent
Default Blocking	Training and Agent bots blocked by default on ad-supported pages from Sept 15, 2026
BotBase	New searchable database of known bots for enterprise customers
Verified Bots	Category now determines access, not just “Verified” status

Improved Bot Management

Previously, Cloudflare offered a simple option to block all AI crawlers. The new system allows website owners to decide whether to permit or restrict bots based on their purpose. “Search” bots, designed for search engine indexing, will generally continue to be allowed. “Training” bots, which collect data for AI model development, and “Agent” bots, which act on behalf of users (like those powering large language models), can now be managed separately.

Impact on Ad-Supported Pages

A notable change effective September 15, 2026, is that Training and Agent bots will be blocked by default on ad-supported web pages. Cloudflare states this is because advertising signals a site’s preference for human visitors. Search crawlers will not be affected by this default block. For multi-purpose crawlers, such as Googlebot, which serve both search and training functions, the stricter rule will apply based on the site’s configuration.

New Tools for Enterprise Customers

Enterprise customers will also benefit from BotBase, a new searchable database integrated into the Cloudflare dashboard. This tool provides detailed information on known bots, including their classification and how they interact with content (e.g., linking to pages versus reproducing full content).

Cloudflare is also revising its policy for verified bots. While previously all verified bots were automatically granted access, their category will now determine access. Bot operators will need to demonstrate honest identification and responsible access practices, moving beyond a simple “Verified” label.

Rationale Behind the Update

Cloudflare CEO Matthew Prince has been vocal about the increasing volume of bot traffic, noting in June 2026 that it had surpassed human traffic on the internet for the first time—a milestone he had not expected until late 2027. Prince has also previously criticized companies like Google for bundling search and AI crawlers, highlighting the need for clearer distinctions. This update reflects Cloudflare’s effort to empower website owners to navigate the evolving landscape of AI and bot interactions more effectively.

Source: The Decoder (https://the-decoder.com/cloudflare-replaces-its-blanket-ai-bot-block-with-granular-controls-for-search-training-and-agent-crawlers/)