AI Referrals Send Minimal Traffic But Double Engagement, Publishers Urged to Rethink Bot Blocking

Publishers who block all AI bots may be missing a significant opportunity. A detailed analysis published on Search Engine Journal by Harry Clarkson-Bennett, SEO Director at The Telegraph, argues that while generative AI platforms send very little direct traffic, the visitors they do send are twice as engaged as average users. The analysis, drawing on Similarweb research and industry data, suggests that publishers who can prove their influence within AI knowledge graphs could build a new revenue stream.

The article arrives at a time when most Indian and global publishers are taking a defensive stance. According to research cited in the piece, 79% of top news sites block AI training bots via `robots.txt`, while only 18% block no AI crawlers at all. Lawsuits between publishers and AI companies have become common, and major outlets such as USA Today, Reuters, and Politico have threatened to block Google outright. Yet the underlying value exchange remains almost non-existent, even as Google reports its most valuable quarter ever.

Key Data

Metric	Value
Top news sites blocking AI bots (robots.txt)	79%
Increase in generative AI app downloads (12 months)	134% to 2.7 billion
Monthly generative AI website visits	Nearly 10 billion
AI-sent user engagement vs. average	Twice as engaged

The data challenges the assumption that AI traffic is worthless. Similarweb’s study on downstream visibility found that generative AI is restructuring the web to be more convenient for users, reducing the cognitive load of finding answers. The majority of users do not click unless forced to, but when they do arrive via AI, they are highly motivated. These users typically discover a brand or product through an AI summary, then later search for it directly – making AI a discovery channel that feeds branded search and direct traffic.

Bot Blocking Creates a Break in the Chain

Clarkson-Bennett notes that strict bot blocking undermines a publisher’s ability to appear in AI training data and real-time retrieval. LLMs are trained on vast datasets, and a brand’s inclusion in the right context at the right time helps the model build an accurate representation of that brand’s authority and relevance. Without being present in the training data, a publisher’s chance of being selected for grounding in AI responses is suppressed. The robots.txt barrier, while easily circumvented, signals a refusal to participate in the ecosystem that may harm long-term visibility.

The analysis points to Reddit as a cautionary tale. Despite being frequently cited in initial AI training data, research by Dan Petrovic shows that Reddit is the most rejected website in AI model selection – filtered out more than any other source. Being present in the data is not enough; the platform must also be perceived as authoritative and useful.

Proving Influence as a New Revenue Line

For publishers willing to unblock bots selectively, the opportunity lies in proving influence. Clarkson-Bennett argues that publishers who can demonstrate they are the most influential brand in a specific vertical (e.g., technology > TVs in a region) become a more valuable proposition for commercial teams. Even probabilistic data, from scaled and consistent prompt tracking, can provide a sellable metric.

The article warns that this work should not be done lightly. Prompt tracking has obvious faults, and without deterministic data, the measurement is imperfect. However, run at sufficient scale, it can yield reasonable outcomes. The top three brands in Rand Fishkin’s study showed relatively high average visibility across diverse prompts, suggesting that consistent effort pays off.

What This Means for Indian Publishers

Indian digital publishers, particularly those covering technology, startups, AI, and SEO, operate in a market where generative AI adoption is surging. The same dynamics apply: younger audiences trust individual creators over brands, favour video and audio, and increasingly rely on AI assistants for research. For Indian publishers, the choice to block or unblock AI crawlers is not binary. It involves weighing direct traffic loss against potential brand discovery through AI.

The analysis recommends that publishers with strict bot-blocking policies reconsider them, perhaps unblocking specific sections of their site that contain original reporting, data analysis, or authoritative content. Commercial teams can use visibility data as marketing collateral to attract advertisers who want to be seen alongside the most influential content in a category.

Limitations and Next Steps

The article acknowledges that the data remains probabilistic. Current measurement tools rely on prompt tracking at scale, which has natural limitations due to the uniqueness of each query. However, the correlation between increasing generative AI traffic, branded search, and direct traffic is strong enough to warrant attention. Publishers should begin by auditing their own AI referral data, reviewing their robots.txt policies, and considering how to capture and package influence metrics for commercial use.

Source: Search Engine Journal – “How Publishers Can Monetize AI Visibility” (https://www.searchenginejournal.com/how-publishers-can-monetise-ai-visibility/583345/)