robots.txt Generator for AI Crawlers
Control which AI bots can crawl your site. Toggle access for GPTBot, ClaudeBot, PerplexityBot, and more -- then copy or download your robots.txt.
AI Crawlers
Traditional Crawlers
# robots.txt generated by AY Rank # https://www.ayrank.com/tools/robots-txt-generator User-agent: * Allow: / # AI crawlers - explicitly allowed User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: /
Want your full site optimized for AI crawlers?
A robots.txt file is just the beginning. Get a comprehensive GEO audit covering structured data, content optimization, and AI citation strategy.
Book a Free GEO AuditWhat are AI Crawlers?
AI crawlers are web bots operated by AI companies to index content for their language models and AI search products. Unlike traditional search crawlers, these bots determine whether your content appears in AI-generated answers.
GPTBot
OpenAI's primary crawler used for training data and retrieval-augmented generation. Allowing GPTBot means your content can be cited in ChatGPT responses and OpenAI's search products.
OAI-SearchBot
OpenAI's dedicated search crawler for SearchGPT / ChatGPT Search. It indexes pages specifically for real-time search results within OpenAI's search experience.
ChatGPT-User
The browsing agent used when a ChatGPT user asks it to visit a specific URL. Blocking this prevents ChatGPT from reading your pages in real-time conversations.
ClaudeBot
Anthropic's crawler for Claude AI. Allowing it ensures your content can be referenced by Claude in its responses and knowledge base.
PerplexityBot
Perplexity AI's crawler that indexes content for its AI-powered search engine. Perplexity directly cites sources, making this valuable for brand visibility.
Google-Extended
Google's crawler specifically for Gemini AI training. Separate from Googlebot, blocking it only affects AI training, not your standard Google Search rankings.
Applebot-Extended
Apple's crawler for Apple Intelligence and Siri AI features. Separate from the standard Applebot used for Siri web results and Safari suggestions.
Googlebot
Google's standard web crawler for Search indexing. Blocking it removes your site from Google Search entirely -- only block if you have a specific reason.
Bingbot
Microsoft's standard web crawler for Bing Search. Also feeds into Microsoft Copilot and Bing Chat, so blocking it affects AI visibility on Microsoft platforms.
Frequently Asked Questions
What are AI crawlers?
AI crawlers are web bots used by AI companies to index and train their models. Unlike traditional search engine crawlers (Googlebot, Bingbot), AI crawlers like GPTBot, ClaudeBot, and PerplexityBot collect content to power AI assistants and AI-generated search results.
Should I allow or block AI crawlers in robots.txt?
If you want your content to appear in AI search results (ChatGPT, Perplexity, Google AI Overviews), you should allow AI crawlers. Blocking them means AI models cannot index your content, reducing your visibility in AI-powered search. Most businesses benefit from allowing AI crawlers.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is used by OpenAI for training and retrieval-augmented generation (RAG), while ChatGPT-User is the bot that browses the web when a ChatGPT user asks it to visit a URL. Allowing GPTBot means your content can be cited in ChatGPT responses; allowing ChatGPT-User lets ChatGPT browse your site in real-time.