Cloudflare switched its default rules for AI crawler access on September 15, 2026. Cloudflare announced the change on July 1, 2026, and TechCrunch reported that it covers new Cloudflare customers, new sites set up by existing customers, and all existing free customers. If you were on a paid plan with sites already configured, your settings stayed as they were unless you opted in. The date has passed, but the risk it created hasn't. The new defaults can block the AI crawlers you want, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, while you think you're only blocking training scrapers. If you added a site or ran a free plan after September 15, check your settings now.
What changed on September 15, 2026?
Cloudflare replaced a binary allow-or-block AI bot setting with three categories. Search crawlers, the kind that index content to answer questions later, stay allowed by default. Agent crawlers, bots acting in real time on behalf of a person, are blocked by default on pages that carry ads. Training crawlers, the kind that harvest content to train or fine-tune a model, are also blocked by default on ad-monetized pages (Help Net Security, July 2, 2026).
That last point matters: this default change only bites on pages that display ads. A SaaS site with no ad units isn't automatically touched by this specific default. A publisher running display ads on most of its content is the group this actually reshapes.
Who does this affect, and who's exempt?
Since September 15, 2026, new Cloudflare customers get the new defaults from day one. Existing customers adding a new site get the new defaults on that site. Existing free-tier customers moved to the new defaults on that date. Paying customers whose sites were already live kept their configuration unless they opted into the new categories themselves. Owners could opt out in Security settings before the date; if you didn't, the defaults now apply.
The mistake that actually costs you AI citations
That's the whole risk in one sentence: trying to keep AI companies from training on your content, and accidentally locking the same crawlers out of citing you in an answer.
Where do you check this in the Cloudflare dashboard, and what else can conflict with it?
Go to Security, then Bots, then AI Scrapers and Crawlers, and review your current setting for each of the three categories: Search, Agent, Training. That's the WAF-level control.
Separately, check your robots.txt file for existing disallow rules on GPTBot, Google-Extended, ClaudeBot, PerplexityBot, CCBot, and anthropic-ai. These are two different layers, and they don't defer to each other. A robots.txt line allowing a crawler doesn't override a Cloudflare block, and a Cloudflare allow doesn't override a robots.txt disallow. Both have to say yes for the crawler to actually get through.
content-signal: search=yes, ai-train=no). That text is declarative, not enforced. Google's John Mueller said on Reddit, [as reported by Search Engine Roundtable on July 6, 2026](https://www.seroundtable.com/google-cloudflare-content-signals-41631.html), that the directive has "no effects whatsoever for any crawler or LLM" and "just adds bloat and future maintenance to your robots.txt file." The WAF-level block covered above genuinely stops the HTTP request and works regardless of what's written in robots.txt. The Content Signals text Cloudflare adds alongside it is a note in a public file. Don't mistake seeing it in your robots.txt for enforcement.
| Site type | Recommended Training default | Recommended Agent default | Why |
|---|---|---|---|
| Ad-monetized content site | Keep the new default (blocked) unless you're paid per crawl | Block by default is fine | The ad-supported model is exactly what the new default protects; opting back to allow gives away training access for nothing unless you're in a paid program |
| SaaS site with no ads | Not automatically affected, but check settings anyway if you're on free tier | Allow if you want AI agents completing tasks on a user's behalf on your product pages | Being cited by ChatGPT or Perplexity when someone asks about your category is worth more than blocking training on marketing pages nobody would train a model on anyway |
| E-commerce with ad placements on category/blog pages | Review page by page, not site-wide, since only ad pages are affected | Allow on product pages where an AI shopping agent needs to complete a purchase; block matches your ad pages | A blanket block can lock out the shopping agents you want completing checkouts while barely denting training access on your ungated product data |
Should you take Cloudflare's Pay Per Crawl deal?
This ties into Cloudflare's Pay Per Crawl program, which TechCrunch reported in July 2026 is evolving into "Pay Per Use", letting publishers charge AI companies "when their content creates value, not just when it's fetched." Cloudflare named Ceramic.ai and You.com as its first partners. Cloudflare CEO Matthew Prince said: "We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training." It's a separate decision from the September 15 default change, and not one to rush into before you've sorted out which crawlers you want reaching your content.
What should you check now?
Log into the Cloudflare dashboard and open Security, Bots, AI Scrapers and Crawlers. Confirm the Search category is set to Allowed. Check the Training category against the table above for your site type, and if it's set to blocked, verify that GPTBot, ClaudeBot, PerplexityBot, and Google-Extended aren't swept into that block through a shared classification. Then pull up your robots.txt and check the same four crawler names for disallow rules that might conflict with whatever you just set in the dashboard.
If your robots.txt needs new rules, our robots.txt generator writes the allow lines for each AI crawler. For why a robots.txt block alone often fails to keep GPTBot out, see why blocking GPTBot in robots.txt doesn't work. A clean robots.txt can still sit behind a Cloudflare block, so check both.
The new defaults took effect on September 15, 2026, for new customers, new sites, and free-tier accounts. The real risk is the classification rule: a crawler serving more than one function follows its most restrictive setting, so blocking Training broadly can silently take Search-function access down with it, including the exact crawlers that would otherwise cite you. Check Cloudflare's Security, Bots, AI Scrapers and Crawlers settings and your robots.txt separately. One allowing a crawler doesn't cancel out the other blocking it.
Frequently asked questions
What did Cloudflare change on September 15, 2026?
Cloudflare replaced its binary allow-or-block AI bot setting with three categories: Search, Agent, and Training. Since September 15, 2026, the new defaults apply to new Cloudflare customers, new sites added by existing customers, and all free-tier customers. Existing paid customers kept their settings on existing sites unless they opted in.
Does this affect my site if I'm already a paying Cloudflare customer?
Not automatically. If your sites were already configured on a paid plan before September 15, 2026, your bot settings stayed as they were. The new defaults apply on their own only to new customers, newly added sites, and free-tier accounts.
Can I accidentally block ChatGPT or Perplexity from citing my site while trying to block AI training?
Yes, and it's the risk worth checking first. Crawlers that serve more than one function are classified by their most restrictive setting. A crawler that does both Search and Training gets blocked if you set Training to blocked, even if you wanted to keep its Search function allowed. Verify GPTBot, ClaudeBot, PerplexityBot, and Google-Extended specifically aren't caught in a blanket Training block.
Where do I check my current AI crawler settings in Cloudflare?
In the Cloudflare dashboard, go to Security, then Bots, then AI Scrapers and Crawlers. Review the setting for each of the three categories: Search, Agent, Training.
Does a robots.txt rule override a Cloudflare AI crawler block, or the other way around?
Neither overrides the other. They're separate layers. A robots.txt allow doesn't override a Cloudflare block, and a Cloudflare allow doesn't override a robots.txt disallow rule. Both have to permit a crawler for it to get through.
Sources: Google: Cloudflare Content Signals Robots.txt Directive Has No Effects (Search Engine Roundtable), Cloudflare's new policy pushes AI companies to pay for publishers' content (TechCrunch), Cloudflare changes AI crawler access rules (Help Net Security)
This post is part of our Technical SEO guide. Related reading: Your robots.txt GPTBot Block Probably Isn't, How to Use AI for SEO, GEO for Hotels and Travel.
Walid founded AY Rank to help businesses dominate AI search. He leads the GEO methodology and oversees client strategy across 50+ cities in Europe, Middle East, and North Africa.
Full Bio →
