Charging AI Bots for Crawl Access: A Visibility Bet for Indian Publishers

Two of the largest web infrastructure companies now offer a way to charge AI bots for crawling. Cloudflare launched its pay-per-request feature for AI crawlers in July 2025, and AWS followed with a similar capability in its Web Application Firewall on June 15, 2026. The mechanism uses the HTTP 402 status code, a technical standard written in 1997 but left unused until now. For Indian publishers, bloggers, and e-commerce sites, the new tools shift a question that used to be binary — allow or block — into a deliberate pricing decision. But the cost of putting a bot behind a paywall may be measured in lost visibility, not just revenue.

The mechanism behind the toll

When an AI crawler requests a page from a website that has enabled pay-per-crawl, the server returns HTTP 402 Payment Required, along with a price and a machine-readable payment manifest. The crawler must either present payment credentials or be denied access. Cloudflare handles the settlement as the merchant of record, while AWS uses Coinbase’s stablecoin settlement via the x402 open standard. Both services sit at the edge layer — the same CDN and firewall that already decide whether a request is allowed. Payment and permission have merged into a single control.

The infrastructure is not limited to the two giants. TollBit runs a bot-and-agent paywall marketplace, and Akamai integrated TollBit and Skyfire into its edge in September 2025, allowing customers to enforce the toll at the network layer without modifying their website code. The x402 standard from Coinbase allows anonymous anonymous bots to pay without an account, which could extend the toll to crawlers that are not easily identified.

The broken bargain behind the paywall

The open web’s original trade ran on a simple promise: let a crawler in, get indexed, and receive traffic in return. AI crawlers, however, broke that deal. According to Cloudflare’s analysis, nearly 80% of AI bot activity is for model training — extraction that takes content without sending back a single visitor. Only a small fraction of AI crawls are for search purpose that could return a citation. The HTTP 402 toll is, in effect, the web’s attempt to renegotiate a bargain that AI crawlers already stopped honouring.

The visibility tradeoff

Paying for crawl access does not automatically mean a site is blocked from all AI surfaces. Adobe’s 2026 data showed that AI-referred traffic to US retailers grew 393% year over year. That number underscores that the AI crawler charging and the AI answer that refers a customer are often the same pipeline. A publisher that charges a bot may collect a few cents per request, but it may also delete itself from the place where its audience now asks questions.

For Indian publishers, the stakes are high. Many rely on organic traffic from Google and from AI-generated answers in tools like Google AI Overviews, Perplexity, and ChatGPT. If a website puts a paywall on the bots that feed those answers, it risks vanishing from the very surfaces that are growing. The decision is not a simple revenue calculation. It is a strategic choice about which AI agents still get to cite, recommend, and link to the site.

Datos clave

Aspect	Detail
Cloudflare launch	July 2025 – first major pay-per-crawl feature for AI crawlers, HTTP 402 status
AWS launch	June 15, 2026 – same capability via WAF Bot Control, Coinbase settlement
AI-referred traffic growth	393% year over year for US retailers (Adobe, 2026)

Who benefits, who loses

The toll makes the most sense for sites with deep, licensable archives — news outlets, reference databases, proprietary data sets. For them, the content is genuinely worth paying for, and distribution does not depend on AI crawlers. A content or commerce website chasing AI visibility, on the other hand, sits on the opposite side. The crawler is the distribution. Shutting it out may bring in small payments but could remove the site from the answers that users now rely on.

The agent-as-visitor question, previously a legal abstraction debated in cases like Amazon v. Perplexity, has now become a configurable setting. AWS and Cloudflare give every website owner the ability to decide which agents are visitors and which are paying customers. The decision is not just about who gets in — it is about who gets to cite the site, and whether the site will still appear in the answers that shape the next wave of search.

Source: Search Engine Journal, “Charging AI Bots Decides Which Agents Can Still Cite You” – https://www.searchenginejournal.com/charging-ai-bots-decides-which-agents-can-still-cite-you/580050/