Can AI crawlers read your site?
Enter a domain. We read its robots.txt and homepage, then show Allowed or Blocked for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, the line that decides it, and the lines to paste to fix it.
Example: www.theguardian.com, checked 14 Nov 2026
- GPTBotAllowed
- ClaudeBotBlocked
- PerplexityBotBlocked
- Google-ExtendedAllowed
robots.txt
79 User-Agent: PerplexityBot... 82 User-agent: ClaudeBot...102 Disallow: /
ClaudeBot and PerplexityBot are named in one group (lines 79 and 82), and line 102 blocks both. GPTBot and Google-Extended fall to the * group, which doesn't block the homepage.
How it works
- 1
Enter a domain
Like example.com. We follow redirects to the real homepage and read robots.txt on that host.
- 2
See four verdicts
Allowed or Blocked per bot, the robots.txt line that decides it (RFC 9309), and any meta robots or X-Robots-Tag rule on the homepage.
- 3
Copy the fix
Pick Allow or Block per bot and copy its lines, or one block for all four.
These four tokens do different jobs. OpenAI says blocking GPTBot opts you out of model training while OAI-SearchBot still handles search. Perplexity says PerplexityBot surfaces and links sites in its answers. One stray Disallow can drop you from an AI answer engine you wanted to be in.
Frequently Asked Questions
How do I check if GPTBot is blocked on my site?
Enter your domain above. The checker reads your robots.txt, finds the group that names GPTBot (or the * group if none does) and shows Allowed or Blocked with the exact line that decides it. It does the same for ClaudeBot, PerplexityBot and Google-Extended in the same run.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says GPTBot crawls content that may be used to train its models, and that a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. Blocking GPTBot is a training choice. Check OAI-SearchBot with the robots.txt tester if search is what you care about.
What is Google-Extended, and does blocking it hurt my Google rankings?
Google-Extended is a robots.txt token that controls whether content Google crawls may be used for Gemini training and grounding. Google says it does not affect a site's inclusion in Google Search and is not a ranking signal. It has no user agent of its own, so only robots.txt can set it.
The checker says Allowed, but the bot still can't crawl my site. Why?
robots.txt is only one gate. A CDN bot setting, a firewall rule or a rate limit can block the real crawler by IP or user agent, and this checker can't see that because it never poses as the bot. Upload a slice of your access log to the AI crawler log checker to see the status codes the real bots got.
Do I need a separate robots.txt group for each AI bot?
Only if you want a bot treated differently from your * group. Under RFC 9309 a group that names a bot replaces the * group for that bot, and groups that name the same bot are merged. Keep one group per bot so the longest-match rule stays easy to read.
How this differs from our other crawler tools
- AI training and grounding checker →The AI training and grounding checker reads one page for Bing AI Training, Bing AI Grounding and Google-Extended. This tool takes a whole domain, checks the four AI bots by name, and gives you a fix snippet.
- robots.txt tester →The robots.txt tester checks one URL for one bot you pick, and lets you test edits before you publish. Use it for deep paths. Use this tool for the one-glance answer on the four AI bots.
Sources
We read each vendor page below on 2026-11-14. Bot tokens and what each bot does come from these pages.
- OpenAI, Overview of OpenAI Crawlersdevelopers.openai.com
- Claude Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
- Perplexity, Perplexity Crawlersdocs.perplexity.ai
- Google, Google's common crawlers (Google-Extended)developers.google.com
- RFC 9309, Robots Exclusion Protocol (IETF, September 2022)www.rfc-editor.org
Want us to fix this for you?
This checks four bots on one homepage. Our Technical SEO work maps robots.txt, CDN bot rules and meta tags across every template, so the AI crawlers you want get in and the ones you don't stay out.