Is this URL allowed or blocked by robots.txt?
Enter a page URL and pick a crawler. We read the site's robots.txt and show Allowed or Blocked, plus the exact line that decided it. Paste a file instead to test edits before you deploy them.
Want every AI crawler at once?
This tester checks one path for one bot. The AI crawler checker in the AY Rank app reads your robots.txt for 12 AI crawlers in one pass.
Enter a URL
Any public page, or a path like /blog/post when you paste a file. Rules match the path and the query string.
Pick a crawler
Choose one of 11 crawlers from Google, Bing, OpenAI, Anthropic, Perplexity, Apple and Common Crawl, or type any other token.
We apply RFC 9309
We find the group for that token, then the longest rule that matches. Allow wins a tie.
You see the deciding line
Allowed or Blocked, with the line number and its text, so you know what to change.
One wrong Disallow line can keep Googlebot or an AI search crawler away from the pages you want found. robots.txt is hard to read by eye because the longest matching rule wins, not the first one. Test the exact URL before and after you change the file.
How this works
- We pick the group whose User-agent line names the crawler's token, ignoring case. Groups that name the same token are merged. If no group names it, the User-agent: * group applies. If neither exists, nothing is blocked. (RFC 9309, section 2.2.1)
- Inside that group, the longest Allow or Disallow path that matches the URL decides. The * and $ characters count toward the length. (RFC 9309, section 2.2.2, and the examples on Google's robots.txt page)
- When an Allow path and a Disallow path match with the same length, Allow wins. Google calls this using the least restrictive rule. (RFC 9309, section 2.2.2)
- A * matches any run of characters, and a $ at the end means the URL must end there. The query string is part of the path. (RFC 9309, section 2.2.3)
- Paths are compared in percent-encoded form, so /café and /caf%C3%A9 are the same path, and %62 is the same as b. Matching is case sensitive. (RFC 9309, section 2.2.2)
- A Disallow line with no path blocks nothing, because Google ignores a rule without a path. The /robots.txt file itself is always allowed. (Google's robots.txt page and RFC 9309, section 2.2.2)
- If robots.txt answers 4xx, every path is allowed. If it answers 5xx or does not answer, every path is blocked. Google handles 429 and 5xx answers differently, and the result says how when that applies. (RFC 9309, sections 2.3.1.3 and 2.3.1.4)
- We follow up to five redirects for robots.txt. After that we treat the file as unavailable, which RFC 9309 allows and Google does. (RFC 9309, section 2.3.1.2)
- We read the first 500 KiB of the file. That is Google's limit and the smallest limit RFC 9309 allows. Lines after it are ignored. (RFC 9309, section 2.5)
- Only User-agent, Allow and Disallow lines decide access. Google says it does not support crawl-delay and that its parser ignores every other field, noindex lines included. Anthropic says its bots do read Crawl-delay.
Sources
We read each page below on 2026-11-11. Every crawler token in the list was checked against the vendor's own page.
- RFC 9309, Robots Exclusion Protocol (IETF, September 2022)https://www.rfc-editor.org/rfc/rfc9309
- Google, How Google interprets the robots.txt specificationhttps://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
- Google, Google's common crawlers (Googlebot, Google-Extended)https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Microsoft Bing, Your content, your choice (Bingbot), October 30, 2026https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
- OpenAI, Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web (ClaudeBot, Claude-SearchBot)https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity, Perplexity crawlers (PerplexityBot)https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Apple, About Applebot (Applebot-Extended)https://support.apple.com/en-us/119829
- Common Crawl, CCBothttps://commoncrawl.org/ccbot
Writing a new file? Build one with the robots.txt generator, then test it here.
Frequently Asked Questions
How do I test whether robots.txt blocks a URL?
Enter the URL, pick the crawler and press Test URL. The tester reads the site's /robots.txt, finds the group for that crawler and shows Allowed or Blocked with the line number that decided it.
Which line wins when Allow and Disallow both match?
The longest matching path wins, whatever order the lines are in. If both paths have the same length, Allow wins. Both rules come from RFC 9309, section 2.2.2.
Does User-agent: * apply to GPTBot if GPTBot has its own group?
No. A crawler follows only the group that names it. The * group applies only when no group names the crawler, and Google says the two are never combined.
What happens if my robots.txt returns a 404 or a 500?
Under RFC 9309, a 4xx answer means crawlers may access everything, and a 5xx answer or no answer means they must assume everything is disallowed. Google stops crawling for 12 hours after a 5xx, then uses its last good copy for up to 30 days.
Can I test changes before I upload them?
Yes. Switch to Paste robots.txt, paste your draft and test any path. It runs in your browser. After a live test, the Edit this robots.txt button loads the fetched file so you can change a line and test again.
Want us to fix this for you?
This tests one URL. Our Technical SEO work audits robots.txt, meta robots tags and crawl paths across every template, so Google and AI crawlers reach the pages you want found.