Nearly four in ten sites that think they've blocked GPTBot in robots.txt haven't blocked anything. HasData tested this directly in July 2026: across 592 sites that disallow GPTBot in their robots.txt, 234 of them (39.5%) still served GPTBot a live HTTP 200 when it actually showed up. The block exists on paper. The server never enforces it.
What did HasData actually measure?
HasData analyzed 10,894 registrable domains in July 2026: the Tranco top 10,000 web-wide list plus 1,148 news publishers pulled in specifically because publishers behave differently from the general web. That full sample answered scope questions, whether a domain sits behind Cloudflare, runs ad tech, publishes an llms.txt file, and what its robots.txt says about AI crawlers.
A smaller 2,096-domain subset (the top 1,000 open-web sites plus every publisher in the sample) got tested for actual enforcement. Each domain received two requests from the same US datacenter IP: one carrying GPTBot's real user-agent string, one carrying a standard Chrome browser's. Same IP, same exit, only the identity string changed. If a site's declared robots.txt policy matched its live behavior, the GPTBot request should get blocked while the browser request goes through. Where it didn't match, that's the gap this study is measuring.
How much of the web actually blocks GPTBot in robots.txt?
Declared blocking splits hard along one line: whether the site is a news publisher. GPTBot is disallowed by 7.9% of the whole-web sample, but by 50.5% of the news-publisher subset, roughly six times the rate. Widen the question to any AI crawler, and it's 10.3% whole web versus 56.4% publishers. Publishers have a direct, visible reason to care: their content is exactly what GPTBot is trained on, and their revenue model already prices out anyone who doesn't pay. Most of the rest of the web, e-commerce, SaaS, docs, blogs, doesn't see AI training as a business threat the same way, and it shows in the numbers.
That part of the study confirms what most people already assumed: publishers block more. The next part is where it gets interesting.
Why do 39.5% of declared GPTBot blocks not actually work?
Of the 592 sites in the enforcement subset that disallow GPTBot in robots.txt, 234 still served GPTBot a live 200 response when tested. That's a declared policy with no server-side enforcement behind it, GPTBot reads the same robots.txt file everyone else does, sees the disallow line, and by OpenAI's own stated policy is supposed to skip the site. Whether GPTBot actually respects that in every case, or whether the origin server would let it through anyway if it ignored the rule, HasData's test can't fully separate, but the practical result is the same either way: the content is reachable.
The rate isn't even consistent between the two groups you'd expect to differ. Open-web sites that declare a GPTBot block ignore their own rule 46.7% of the time. Publishers, despite blocking far more often in the first place, still fail to enforce that block 38.9% of the time. Blocking more often doesn't mean enforcing better.
What happens when a site does actually enforce a GPTBot block?
Enforcement, when it exists, splits into more categories than a simple block. Across the tested subset, only 22.2% of enforcement responses are a hard block, HTTP 403, 401, or 451. Another 10.9% return HTTP 402, Cloudflare's Payment Required signal for its pay-per-crawl program, meaning the site let GPTBot's request through but is trying to charge for it rather than deny it outright. The rest are challenges, JavaScript interstitials, rate limiting, or a CAPTCHA. One in ten "blocks" in this data isn't a block at all. It's a bot paywall wearing an HTTP status code.
| Enforcement outcome | Share of enforcement responses | What it actually means |
|---|---|---|
| Hard block (403/401/451) | 22.2% | Server genuinely refuses the request |
| HTTP 402 Payment Required | 10.9% | Cloudflare pay-per-crawl signal, access gated behind payment rather than denied |
| Declared block, live 200 served anyway | 39.5% (of the 592 GPTBot-disallow sites) | Robots.txt says no, server says yes |
| Silent enforcement, no robots.txt rule | 5.5% (115 of 2,096 tested) | WAF/CDN blocks GPTBot with no declared policy behind it |
Can a site block GPTBot without ever writing a robots.txt rule?
Yes, and HasData found it running the other direction too. 115 sites, 5.5% of the tested subset, blocked GPTBot at the infrastructure or WAF level despite having no robots.txt disallow rule for it at all. Their declared policy is silent or permissive. Their actual behavior is a block. Some hosting and CDN configurations filter known AI crawler user-agents by default, and the site owner may never have set that policy directly, or may not even know it's there.
Put both directions together and the message is the same: what a robots.txt file says and what a server actually does are two different systems that don't automatically agree with each other. One can say yes while the other says no, in either direction.
Our AI Visibility Checker reads exactly the declared side of this, whether your robots.txt currently disallows GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, alongside structured data and content structure. It tells you what your robots.txt says. It doesn't send a live GPTBot-versus-browser request pair the way HasData's enforcement test did, so it can't confirm your server actually behaves the way that file claims. A clean scan there rules out the declared-policy half of this problem, not the enforcement half.
So does robots.txt actually matter here?
It's still the first thing to check, it's just not the last. If you want GPTBot out for real, for training-data, IP, or licensing reasons, a disallow line only works if something on the server side backs it up: a WAF rule, a CDN-level filter (this is exactly what Cloudflare's own crawler-category settings, covered in our September 15 deadline piece, do at the infrastructure layer), or a verified test showing the crawler is actually refused. If your goal runs the other way, getting cited, a leftover disallow rule that nobody's enforcing might not even be costing you anything in practice, but it's still worth removing so it isn't quietly doing damage the day a crawler that does respect it comes along. Either way, checking the file and checking actual behavior are two separate steps, and skipping the second one is how 39.5% of declared bans ended up doing nothing.
HasData tested 592 sites that disallow GPTBot in robots.txt against live requests and found 234 (39.5%) still served GPTBot anyway, a declared block with no enforcement behind it. Publishers block GPTBot roughly six times more often than the open web (50.5% vs 7.9%) but still fail to enforce their own rule 38.9% of the time. Even where enforcement is real, only 22.2% is a hard block, another 10.9% is a Cloudflare pay-per-crawl paywall rather than a denial. A robots.txt line is a declared policy, not a technical control, verify actual crawler behavior separately if the distinction matters to you.
Frequently asked questions
What did HasData's AI Crawler Block Index actually find?
Testing 592 sites that disallow GPTBot in their robots.txt, HasData found 234 of them (39.5%) still served GPTBot a live HTTP 200 response when actually crawled. The declared block in robots.txt didn't match the site's real server behavior.
Does blocking GPTBot in robots.txt actually stop it from crawling?
Not reliably on its own. Robots.txt is a declared policy that a well-behaved crawler is expected to read and follow, it isn't a technical control the server enforces. HasData's data shows 39.5% of sites with a declared GPTBot disallow still served GPTBot a live 200 when tested directly.
Do news publishers block AI crawlers more than other websites?
Yes, by a wide margin. Publishers disallow GPTBot in robots.txt at 50.5% versus 7.9% for the whole-web sample, roughly six times the rate. For blocking any AI crawler, it's 56.4% versus 10.3%.
What is an HTTP 402 response from an AI crawler block, and why does it matter?
HTTP 402 is Payment Required, and in this context it's Cloudflare's pay-per-crawl signal. Of sites that actually enforce something against GPTBot, 10.9% return a 402 rather than a hard block, meaning the crawler is let through in exchange for payment rather than denied outright. Only 22.2% of enforcement responses are a genuine hard block (403, 401, or 451).
Can a site block GPTBot without having any robots.txt rule against it?
Yes. HasData found 115 sites (5.5% of the tested subset) that blocked GPTBot at the infrastructure or WAF level with no robots.txt disallow rule for it at all. Some hosting and CDN configurations filter AI crawler user-agents by default, independent of what the site's robots.txt declares.
Does AY Rank's AI Visibility Checker test actual GPTBot enforcement?
No. It checks what your robots.txt currently declares for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, along with structured data and content structure. It doesn't send a live paired request from a real crawler user-agent versus a browser user-agent, so it can't confirm whether your server actually behaves the way your robots.txt claims.
Source: HasData, "AI Crawler Block Index", July 2026, 10,894 domains analyzed (9,746 Tranco top-web + 1,148 news publishers), 2,096-domain enforcement subset.
This post is part of our Technical SEO guide. Related reading: Schema markup doesn't move AI citations, Cloudflare's AI Crawler Rules Change September 15, Free SEO Audit Template for 2026.
Oussama leads technical and on-page SEO at AY Rank. He specializes in structured data engineering, crawl optimization, and building the entity architecture that makes AI models cite our clients.
Full Bio →
