Common robots.txt Mistakes to Avoid
1. Using a wildcard block without exceptions
# WRONG, blocks Googlebot too
User-agent: *
Disallow: /
If you use a wildcard Disallow, you must explicitly allow each bot you want to permit. Many site owners inadvertently block Googlebot this way.
2. Confusing training bots with search bots
Blocking GPTBot (training) is different from blocking OAI-SearchBot (search retrieval). Blocking the wrong one achieves neither your training-opt-out goal nor your GEO visibility goal.
3. Forgetting anthropic-ai alongside ClaudeBot
Anthropic uses both ClaudeBot and anthropic-ai as user-agent strings. A robots.txt that only mentions one will not fully restrict the other.
4. Placing specific bot rules before the wildcard
Some parsers handle ordering differently. Best practice is: wildcard rule first, then specific bot rules. Specific rules always override the wildcard regardless of order, but this convention makes your file easier to audit.
5. Not including a Sitemap directive
The Sitemap: line at the bottom of robots.txt is how AI crawlers discover your content index. Omitting it is a missed opportunity.
6. Blocking /api/ paths that serve public content
Some Next.js and headless CMS sites serve rendered content via API routes. If AI crawlers are blocked from those paths, they may not access the content even if the marketing pages are allowed.
7. Inconsistent www vs. non-www
rob.txt at www.domain.com/robots.txt applies only to the www subdomain. domain.com/robots.txt covers the apex domain. Most sites should have matching content at both.
8. Declaring Disallow before Allow in the same block
# WRONG order
User-agent: GPTBot
Disallow: /
Allow: /blog/
In many parsers, Disallow: / catches everything before Allow: /blog/ is processed. Use the most specific rule first:
# CORRECT order
User-agent: GPTBot
Allow: /blog/
Disallow: /
How to Test Your robots.txt
Google Search Console Robots.txt Tester
Google Search Console (search.google.com/search-console) includes a built-in tester. Submit any URL on your domain and it will tell you whether Googlebot can or cannot access it based on your current robots.txt. While it only tests Googlebot and Bingbot natively, it is the fastest way to catch syntax errors.
Fetch and Inspect Manually
curl -s https://yourdomain.com/robots.txt
Confirm the file returns HTTP 200 (not 301, 302, or 404). A redirect on robots.txt can confuse crawlers.
Online Robots.txt Validators
- robots-checker.io, validates syntax and simulates bot access
- technicalseo.com/tools/robots-txt-tester, per-bot simulation
- AY Rank Robots.txt Generator, /tools/robots-txt-generator, generate and validate AI-optimized configurations
- AY Rank robots.txt Tester, /tools/robots-txt-tester, checks one URL for any user agent and shows the line that decided it
Check Actual Bot Behaviour
In your web server access logs, filter for AI bot user-agent strings:
grep -i "GPTBot\|OAI-SearchBot\|ClaudeBot\|PerplexityBot\|Google-Extended" /var/log/nginx/access.log
If blocked bots continue to appear after you set Disallow: /, they are either:
(a) not complying with the protocol (common with some scrapers), or
(b) still processing a cached version of your robots.txt (wait 24-48 hours for re-crawl).
Anyone can send a GPTBot or ClaudeBot user agent, so check which hits came from the vendor's own IP ranges with our AI crawler log checker.
If the bots you allow are crawling too hard rather than ignoring you, robots.txt is a slow fix. Google documents a short-term brake instead: return 503 or 429, with Retry-After, to crawler requests for a few hours, never more than 1-2 days. Our guide to slowing down Googlebot and AI crawlers with 429 and Retry-After has tested nginx, Cloudflare Worker and Next.js configs, and what each AI crawler vendor documents about those codes.
robots.txt and llms.txt: Using Both Together
The emerging llms.txt standard (proposed by Answer.AI) is a companion file, not a replacement for robots.txt. While robots.txt controls crawler access, llms.txt provides curated guidance, a structured summary of your most important content for LLMs to use.
A complete AI-visibility setup uses both:
- robots.txt, access control layer (allow/block crawlers)
- llms.txt, curation layer (direct LLMs to your best content)
- Schema.org structured data, context layer (help LLMs understand what your content is about)
For the full checklist, including server log monitoring and JS-rendering fixes, see our guide on how to make your website readable by AI agents.
robots.txt for Common CMS Platforms
Next.js App Router
Next.js 13+ supports a robots.ts file at src/app/robots.ts:
import { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: '*', allow: '/' },
{ userAgent: 'GPTBot', disallow: '/' },
{ userAgent: 'ClaudeBot', disallow: '/' },
{ userAgent: 'Google-Extended', disallow: '/' },
{ userAgent: 'OAI-SearchBot', allow: '/' },
{ userAgent: 'PerplexityBot', allow: '/' },
],
sitemap: 'https://yourdomain.com/sitemap.xml',
}
}
WordPress
WordPress generates a virtual robots.txt. Use a plugin like Yoast SEO or Rank Math to edit it, or add a physical robots.txt file to your server root (which overrides the virtual one).
Shopify
Shopify does not allow direct editing of robots.txt in standard plans. Use the Liquid-based robots.txt.liquid template (available in Online Store > Themes > Edit Code) to customise directives.
Monitoring and Updating Your robots.txt
AI crawler user-agent strings change as platforms launch new products. Build a quarterly review into your SEO workflow:
- Check operator documentation, OpenAI, Anthropic, Google, and Perplexity all publish their current bot names
- Review your access logs for new bot strings you do not recognise
- Monitor AI referral traffic in GA4, unexpected drops may indicate an accidental block
- Re-validate after CMS updates, platform updates sometimes regenerate or overwrite robots.txt
A correctly configured robots.txt only gets a crawler to your page. It does not make client-rendered widgets (booking calendars, pricing tables, FAQ accordions) readable once it arrives; see why a JS-only booking widget is invisible to AI search for that separate, and just as common, gap.
If your site sits behind Cloudflare, read what changed with the Cloudflare AI crawler block.
Frequently Asked Questions
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot collects data to train OpenAI's models. Appearing in ChatGPT search results is controlled separately by OAI-SearchBot, which crawls for real-time retrieval, and ChatGPT-User, which fires during live browsing. You can block GPTBot to opt out of training and still allow the other two, so ChatGPT can keep surfacing and citing your pages.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended only controls whether your content trains Gemini and future Google AI models. AI Overviews are driven by standard Googlebot indexing, and robots.txt cannot control them separately. Applebot-Extended works the same way for Apple: blocking it limits Apple Intelligence training without affecting Siri search visibility.
Which AI crawlers should I allow for GEO?
The most important bots for GEO visibility are OAI-SearchBot, PerplexityBot and Googlebot. PerplexityBot matters because Perplexity is a search-first product, and blocking its crawler removes your site from Perplexity results entirely. For maximum AI visibility, allow all search-oriented bots and block only pure training scrapers if training use concerns you.
Do I need to block both ClaudeBot and anthropic-ai?
Yes. Anthropic uses two user-agent strings: ClaudeBot, which trains Claude models, and anthropic-ai, a legacy string from the same operator. A robots.txt that mentions only one will not fully restrict the other, so include both in any allow or block directive you write for Anthropic.
Can robots.txt stop AI companies from using my content?
Only partly. robots.txt works on the honour system, so compliant crawlers respect it and determined bad actors can ignore it. It cannot remove content from models that already ingested it or pull cached content out of existing training sets, and it is no substitute for server-side authentication on sensitive content. Some researchers have reported Bytespider ignoring robots.txt directives.
How do I check whether AI bots respect my robots.txt?
Fetch the file with curl and confirm it returns HTTP 200, not a redirect or a 404. Then filter your server access logs for AI user-agent strings such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. If a blocked bot keeps appearing, it is either ignoring the protocol or still using a cached copy of your robots.txt, so allow 24 to 48 hours for a re-crawl.
Key Takeaways
- AI crawlers fall into two categories: search-retrieval bots (drive traffic, allow these) and training scrapers (ingest data for model training, allow or block based on your content strategy)
- The most important bots for GEO visibility are
OAI-SearchBot, PerplexityBot, and Googlebot
- Block training scrapers with targeted
Disallow: / rules under specific User-agent: blocks, not with a wildcard
- Always include
Applebot-Extended and Google-Extended in your strategy, they are training-only bots that are completely separate from search indexing
- Test your configuration with Google Search Console and manual log analysis
- Use our Robots.txt Generator to build a validated, AI-optimised configuration in under two minutes