Free Tool

Can AI crawlers read your site?

Enter a domain. We read its robots.txt and homepage, then show Allowed or Blocked for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, the line that decides it, and the lines to paste to fix it.

Example: ⁦www.theguardian.com⁩, checked 14 Nov 2026

  • GPTBotAllowed
  • ClaudeBotBlocked
  • PerplexityBotBlocked
  • Google-ExtendedAllowed

robots.txt

 79  User-Agent: PerplexityBot... 82  User-agent: ClaudeBot...102  Disallow: /

ClaudeBot and PerplexityBot are named in one group (lines 79 and 82), and line 102 blocks both. GPTBot and Google-Extended fall to the * group, which doesn't block the homepage.

How it works

  1. 1

    Enter a domain

    Like example.com. We follow redirects to the real homepage and read robots.txt on that host.

  2. 2

    See four verdicts

    Allowed or Blocked per bot, the robots.txt line that decides it (RFC 9309), and any meta robots or X-Robots-Tag rule on the homepage.

  3. 3

    Copy the fix

    Pick Allow or Block per bot and copy its lines, or one block for all four.

These four tokens do different jobs. OpenAI says blocking GPTBot opts you out of model training while OAI-SearchBot still handles search. Perplexity says PerplexityBot surfaces and links sites in its answers. One stray Disallow can drop you from an AI answer engine you wanted to be in.

Frequently Asked Questions

How do I check if GPTBot is blocked on my site?

Enter your domain above. The checker reads your robots.txt, finds the group that names GPTBot (or the * group if none does) and shows Allowed or Blocked with the exact line that decides it. It does the same for ClaudeBot, PerplexityBot and Google-Extended in the same run.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says GPTBot crawls content that may be used to train its models, and that a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. Blocking GPTBot is a training choice. Check OAI-SearchBot with the robots.txt tester if search is what you care about.

What is Google-Extended, and does blocking it hurt my Google rankings?

Google-Extended is a robots.txt token that controls whether content Google crawls may be used for Gemini training and grounding. Google says it does not affect a site's inclusion in Google Search and is not a ranking signal. It has no user agent of its own, so only robots.txt can set it.

The checker says Allowed, but the bot still can't crawl my site. Why?

robots.txt is only one gate. A CDN bot setting, a firewall rule or a rate limit can block the real crawler by IP or user agent, and this checker can't see that because it never poses as the bot. Upload a slice of your access log to the AI crawler log checker to see the status codes the real bots got.

Do I need a separate robots.txt group for each AI bot?

Only if you want a bot treated differently from your * group. Under RFC 9309 a group that names a bot replaces the * group for that bot, and groups that name the same bot are merged. Keep one group per bot so the longest-match rule stays easy to read.

How this differs from our other crawler tools

Sources

We read each vendor page below on 2026-11-14. Bot tokens and what each bot does come from these pages.

Want us to fix this for you?

This checks four bots on one homepage. Our Technical SEO work maps robots.txt, CDN bot rules and meta tags across every template, so the AI crawlers you want get in and the ones you don't stay out.