Cloudflare ships granular AI bot controls for Search, Agent, and Training traffic
Cloudflare launched new site-owner tools that classify bots as Search, Agent, or Training traffic and let operators set different access rules for each class. The update also helps protect ad-monetized pages from AI crawlers that scrape content without generating impressions.
Key takeaway
AI crawling is moving from an all-or-nothing blocklist problem into a commercial access-control layer where publishers can prefer search indexing while throttling training and agent scrapers.
Context
Until now, many sites treated AI crawlers as a binary allow-or-block choice, which either starved legitimate search discovery or left training and agent bots free to consume content. Cloudflare's Content Independence Day options split that traffic into labeled categories so operators can write differentiated policies instead of a single blanket rule.
That matters for the AI supply chain because publishers can now defend ad-funded pages and subscription surfaces without fully cutting themselves out of search. For model builders and agent vendors, it raises the cost of silent scraping and pushes traffic toward negotiated or otherwise authorized access paths.