Cloudflare is switching the default rules for AI crawler access on September 15, 2026, nine days from now. The change applies to new Cloudflare customers, any new site an existing customer adds, and every free-tier account. If you're on a paid plan with sites already configured, your current settings hold unless you opt into the new defaults yourself. The practical risk isn't the deadline. It's that the new default can block the exact AI crawlers you want, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, while you think you're only blocking training scrapers.
What is actually changing on September 15?
Cloudflare is replacing a binary allow-or-block AI bot setting with three categories. Search crawlers, the kind that index content to answer questions later, stay allowed by default. Agent crawlers, bots acting in real time on behalf of a person, get blocked by default on pages that carry ads. Training crawlers, the kind that harvest content to train or fine-tune a model, also get blocked by default on ad-monetized pages.
That last point matters: this default change only bites on pages that display ads. A SaaS site with no ad units isn't automatically touched by this specific default. A publisher running display ads on most of its content is the group this actually reshapes.
Who does this affect, and who's exempt?
New Cloudflare customers get the new defaults from day one. Existing customers adding a new site get the new defaults on that site. Every free-tier customer, new or existing, moves to the new defaults regardless of when their account started. If you're already a paying customer with sites live on Cloudflare today, nothing changes on your existing configuration unless you go in and opt into the new categories yourself.
The mistake that actually costs you AI citations
That's the whole risk in one sentence: trying to keep AI companies from training on your content, and accidentally locking the same crawlers out of citing you in an answer.
Where do you check this in the Cloudflare dashboard, and what else can conflict with it?
Go to Security, then Bots, then AI Scrapers and Crawlers, and review your current setting for each of the three categories: Search, Agent, Training. That's the WAF-level control.
Separately, check your robots.txt file for existing disallow rules on GPTBot, Google-Extended, ClaudeBot, PerplexityBot, CCBot, and anthropic-ai. These are two different layers, and they don't defer to each other. A robots.txt line allowing a crawler doesn't override a Cloudflare block, and a Cloudflare allow doesn't override a robots.txt disallow. Both have to say yes for the crawler to actually get through.
content-signal: search=yes, ai-train=no). That text is declarative, not enforced. Google's John Mueller said on Reddit on July 6, 2026 that the directive "has no effects whatsoever for any crawler or LLM" and that, as far as he's aware, no crawler or LLM actually reads it. The WAF-level block covered above genuinely stops the HTTP request and works regardless of what's written in robots.txt. The Content Signals text Cloudflare adds alongside it is closer to a note left in a public file that nothing on the other end is currently reading, don't mistake seeing it in your robots.txt for actual enforcement.
| Site type | Recommended Training default | Recommended Agent default | Why |
|---|---|---|---|
| Ad-monetized content site | Keep the new default (blocked) unless you're paid per crawl | Block by default is fine | The ad-supported model is exactly what the new default protects; opting back to allow gives away training access for nothing unless you're in a paid program |
| SaaS site with no ads | Not automatically affected, but check settings anyway if you're on free tier | Allow if you want AI agents completing tasks on a user's behalf on your product pages | Being cited by ChatGPT or Perplexity when someone asks about your category is worth more than blocking training on marketing pages nobody would train a model on anyway |
| E-commerce with ad placements on category/blog pages | Review page by page, not site-wide, since only ad pages are affected | Allow on product pages where an AI shopping agent needs to complete a purchase; block matches your ad pages | A blanket block can lock out the shopping agents you want completing checkouts while barely denting training access on your ungated product data |
Should you take Cloudflare's Pay Per Crawl deal?
This ties into Cloudflare's Pay Per Use program, where publishers can get paid when an AI company's crawler uses their content. It started as a flat per-fetch charge and is moving toward compensation tied to whether the content actually shows up in an AI-generated answer. Ceramic.ai and You.com are named as early partners in press coverage. Cloudflare CEO Matthew Prince framed the shift this way: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." Worth knowing about, but it's a separate decision from the September 15 default change, and not one to rush into before you've sorted out which crawlers you actually want reaching your content.
What should you check before September 15?
Log into the Cloudflare dashboard and open Security, Bots, AI Scrapers and Crawlers. Confirm the Search category is set to Allowed. Check the Training category against the table above for your site type, and if it's set to blocked, verify that GPTBot, ClaudeBot, PerplexityBot, and Google-Extended aren't swept into that block through a shared classification. Then pull up your robots.txt and check the same four crawler names for disallow rules that might conflict with whatever you just set in the dashboard.
Our AI Visibility Checker checks the robots.txt side of this specifically: whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are allowed to crawl, alongside structured data and content structure. It doesn't read Cloudflare's WAF-level bot settings, that's a separate system from robots.txt, so it can tell you your robots.txt is clean while a Cloudflare block is still quietly doing the work. Check both.
The deadline is September 15, 2026, and it only touches new customers, new sites, and free-tier accounts by default. The real risk is the classification rule: a crawler serving more than one function follows its most restrictive setting, so blocking Training broadly can silently take Search-function access down with it, including the exact crawlers that would otherwise cite you. Check Cloudflare's Security, Bots, AI Scrapers and Crawlers settings and your robots.txt separately. One allowing a crawler doesn't cancel out the other blocking it.
Frequently asked questions
What is Cloudflare changing on September 15, 2026?
Cloudflare is replacing its binary allow-or-block AI bot setting with three categories: Search, Agent, and Training. The new defaults apply to new Cloudflare customers, new sites added by existing customers, and all free-tier customers. Existing paid customers keep their current settings on existing sites unless they opt in.
Does this affect my site if I'm already a paying Cloudflare customer?
Not automatically. If your sites are already configured on a paid plan, your current bot settings stay as they are. The new defaults only apply on their own to new customers, newly added sites, and free-tier accounts.
Can I accidentally block ChatGPT or Perplexity from citing my site while trying to block AI training?
Yes, and this is the risk worth checking before September 15. Crawlers that serve more than one function are classified by their most restrictive setting. A crawler that does both Search and Training gets blocked if you set Training to blocked, even if you wanted to keep its Search function allowed. Verify GPTBot, ClaudeBot, PerplexityBot, and Google-Extended specifically aren't caught in a blanket Training block.
Where do I check my current AI crawler settings in Cloudflare?
In the Cloudflare dashboard, go to Security, then Bots, then AI Scrapers and Crawlers. Review the setting for each of the three categories: Search, Agent, Training.
Does a robots.txt rule override a Cloudflare AI crawler block, or the other way around?
Neither overrides the other. They're separate layers. A robots.txt allow doesn't override a Cloudflare block, and a Cloudflare allow doesn't override a robots.txt disallow rule. Both have to permit a crawler for it to get through.
Does the AI Visibility Checker check Cloudflare's bot settings?
No. It checks robots.txt for crawler access rules covering GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, along with structured data and content structure. Cloudflare's WAF-level AI Scrapers and Crawlers settings are a separate system and need to be checked directly in the Cloudflare dashboard.
Sources: Cloudflare's new policy pushes AI companies to pay for publishers' content (TechCrunch), Cloudflare changes AI crawler access rules (Help Net Security)
This post is part of our Technical SEO guide. Related reading: Free SEO Audit Template for 2026, Schema Markup for GEO, Structured Data Engineering for AI Citation.
Walid founded AY Rank to help businesses dominate AI search. He leads the GEO methodology and oversees client strategy across 50+ cities in Europe, Middle East, and North Africa.
Full Bio →
