Blog
4 August 2026/12 min read

robots.txt for AI Crawlers: Complete Setup Guide

A complete guide to configuring robots.txt for every major AI crawler, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more. Learn which bots to allow, which to block, and how to test your configuration.

Run a free check on app.ayrank.comNo sign-up. See if ChatGPT names your brand.NeoDayClient result: NeoDay grew AI citations 340% in 6 months.
Abdelmoghit Idhsaine
Author:Abdelmoghit Idhsaine,Content Strategist
robots.txt for AI Crawlers: Complete Setup Guide

robots.txt for AI Crawlers: Complete Setup Guide

Your robots.txt file is no longer just a directive for Googlebot and Bingbot. In 2026 it has become the primary access-control layer for a new generation of AI crawlers, systems that feed ChatGPT, Claude, Perplexity, Google AI Overviews, Apple Intelligence, and dozens of other large-language-model products.

Getting this file wrong has real business consequences: you may accidentally block crawlers that drive referral traffic, or leave your content exposed to scrapers that train competing AI models without compensating you.

This guide covers every major AI crawler, explains what each one does, and gives you a production-ready robots.txt template you can deploy today.


What robots.txt Actually Does for AI Crawlers

The Robots Exclusion Protocol is a plain-text convention, not a legal contract or enforced standard. It works on the honour system: you publish directives, and compliant crawlers respect them. The key word is compliant.

What robots.txt can do:

  • Signal to well-behaved AI crawlers which pages or paths to avoid
  • Distinguish between indexing bots and training-data scrapers
  • Grant or deny access on a per-bot-name basis
  • Set crawl-delay hints to protect server resources

What robots.txt cannot do:

  • Block determined bad actors who ignore the protocol
  • Prevent your already-cached content from being used in existing AI training sets
  • Remove content from AI models that already ingested it
  • Substitute for robust server-side authentication on sensitive content

With those boundaries clear, a well-crafted robots.txt is still your first line of defence and the easiest lever you have.


The Complete List of AI Crawlers (2026)

The AI crawler landscape has grown rapidly. Here is every significant bot you need to know about, organized by operator.

OpenAI

Bot NamePurpose
GPTBotTrains OpenAI's language models (GPT-4, GPT-4o, etc.)
OAI-SearchBotPowers real-time web search inside ChatGPT
ChatGPT-UserIssued when a ChatGPT user triggers a live browsing request

GPTBot is the big one. If you block it, your content will not be used to improve future OpenAI models. However, blocking GPTBot does not prevent your site from appearing in ChatGPT search results, that is controlled separately by OAI-SearchBot and ChatGPT-User.

OAI-SearchBot crawls for real-time retrieval. Blocking it means ChatGPT cannot surface your pages when users ask questions that would otherwise cite you. For most content marketers this is traffic you want.

ChatGPT-User fires per user session during live browsing. It typically respects the same rules as OAI-SearchBot.

Anthropic (Claude)

Bot NamePurpose
ClaudeBotTrains Anthropic's Claude models
anthropic-aiLegacy user-agent string, same operator

Anthropic uses two different user-agent strings. To be safe, include both in any allow or block directive.

Perplexity AI

Bot NamePurpose
PerplexityBotPowers Perplexity's real-time AI search

Perplexity is a search-first product, it surfaces citations to users actively searching. Blocking PerplexityBot removes your site from Perplexity results entirely. For GEO-focused strategies, this bot is one of the highest-priority crawlers to allow.

Google

Bot NamePurpose
Google-ExtendedTrains Google's Gemini and future AI models
GooglebotCore web indexing (also used for AI Overviews)

Google-Extended was introduced specifically to give publishers control over AI training separately from search indexing. Blocking Google-Extended will not remove you from AI Overviews, that is driven by standard Googlebot indexing and is not separately controllable via robots.txt.

Apple

Bot NamePurpose
ApplebotPowers Siri, Spotlight, and general Apple web indexing
Applebot-ExtendedTrains Apple Intelligence AI models

Same logic as Google: blocking Applebot-Extended limits AI training without affecting Siri search visibility.

Meta

Bot NamePurpose
FacebookBotCrawls for link previews and Meta AI training
meta-externalagentMeta's newer AI-specific agent

Facebook link previews are driven by FacebookBot. If you block it entirely, your URLs will show blank previews when shared on Facebook and Instagram.

Cohere

Bot NamePurpose
cohere-aiTrains Cohere's enterprise language models

Cohere primarily serves B2B enterprise AI products. Less likely to drive direct referral traffic.

ByteDance / TikTok

Bot NamePurpose
BytespiderPowers ByteDance products including TikTok search

Bytespider has a mixed reputation for protocol compliance. Some security researchers have reported it ignoring robots.txt directives. Blocking it is a reasonable precaution if you are concerned about scraping.

Other Notable Bots

Bot NameOperatorPurpose
CCBotCommon CrawlOpen dataset used to train many open-source LLMs
DataForSeoBotDataForSEOSEO data aggregator
YouBotYou.comAI search engine
DiffbotDiffbotKnowledge graph / AI training
AI2BotAllen Institute for AIResearch AI training

How to Allow or Block Each Crawler

The syntax is straightforward. Each block begins with a User-agent: line followed by Allow: or Disallow: directives.

Block a specific bot entirely

User-agent: GPTBot
Disallow: /

Allow a specific bot to access everything

User-agent: PerplexityBot
Allow: /

Block a bot from specific paths only

User-agent: GPTBot
Disallow: /private/
Disallow: /members/
Allow: /blog/
Allow: /

Allow a bot despite a broader Disallow rule

User-agent: *
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Important: Bot-specific blocks take precedence over the wildcard * block. Declare specific agents after the wildcard block, not before.


Before writing your robots.txt, decide what you actually want from AI platforms:

Goal A: Maximum AI search visibility (GEO-first strategy) Allow all search-oriented bots. Block only pure training scrapers if you have concerns about AI training use.

Goal B: Block AI training, preserve search Allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot. Block GPTBot, ClaudeBot, Google-Extended, anthropic-ai, cohere-ai, Bytespider.

Goal C: Block all AI activity Block every AI bot. This is rarely advisable for marketing sites as it removes you from an increasingly significant traffic source.

Goal D: Selective by content type Allow bots on public marketing content, block on proprietary research or gated resources.


Complete Production robots.txt Template

Here is a fully-annotated template. Remove the comment lines before deploying.

# =============================================================
# robots.txt for [yourdomain.com]
# Last updated: 2026-05-12
# Strategy: Allow AI search bots, block AI training scrapers
# =============================================================

# --- Standard search engines (always allow) ---
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Slurp
Allow: /

User-agent: DuckDuckBot
Allow: /

# --- OpenAI: allow search, block training ---
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

# --- Anthropic: allow search, block training ---
# (Claude does not yet have a separate search bot)
User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

# --- Perplexity: allow (search-first product) ---
User-agent: PerplexityBot
Allow: /

# --- Google AI training (separate from Googlebot) ---
User-agent: Google-Extended
Disallow: /

# --- Apple: allow Siri search, block AI training ---
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

# --- Meta: allow link previews, review training ---
User-agent: FacebookBot
Allow: /

User-agent: meta-externalagent
Disallow: /

# --- Cohere: block training ---
User-agent: cohere-ai
Disallow: /

# --- ByteDance / TikTok ---
User-agent: Bytespider
Disallow: /

# --- Common Crawl (used for open LLM training) ---
User-agent: CCBot
Disallow: /

# --- You.com AI search ---
User-agent: YouBot
Allow: /

# --- Allen Institute ---
User-agent: AI2Bot
Disallow: /

# --- Default: allow everything not covered above ---
User-agent: *
Allow: /

# --- Sitemap location (always include) ---
Sitemap: https://yourdomain.com/sitemap.xml

Template for maximum AI search visibility (GEO-first)

If your goal is citation and referral traffic from every AI platform, use this simpler version:

# Allow all AI crawlers for maximum GEO visibility
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Common robots.txt Mistakes to Avoid

1. Using a wildcard block without exceptions

# WRONG, blocks Googlebot too
User-agent: *
Disallow: /

If you use a wildcard Disallow, you must explicitly allow each bot you want to permit. Many site owners inadvertently block Googlebot this way.

2. Confusing training bots with search bots

Blocking GPTBot (training) is different from blocking OAI-SearchBot (search retrieval). Blocking the wrong one achieves neither your training-opt-out goal nor your GEO visibility goal.

3. Forgetting anthropic-ai alongside ClaudeBot

Anthropic uses both ClaudeBot and anthropic-ai as user-agent strings. A robots.txt that only mentions one will not fully restrict the other.

4. Placing specific bot rules before the wildcard

Some parsers handle ordering differently. Best practice is: wildcard rule first, then specific bot rules. Specific rules always override the wildcard regardless of order, but this convention makes your file easier to audit.

5. Not including a Sitemap directive

The Sitemap: line at the bottom of robots.txt is how AI crawlers discover your content index. Omitting it is a missed opportunity.

6. Blocking /api/ paths that serve public content

Some Next.js and headless CMS sites serve rendered content via API routes. If AI crawlers are blocked from those paths, they may not access the content even if the marketing pages are allowed.

7. Inconsistent www vs. non-www

rob.txt at www.domain.com/robots.txt applies only to the www subdomain. domain.com/robots.txt covers the apex domain. Most sites should have matching content at both.

8. Declaring Disallow before Allow in the same block

# WRONG order
User-agent: GPTBot
Disallow: /
Allow: /blog/

In many parsers, Disallow: / catches everything before Allow: /blog/ is processed. Use the most specific rule first:

# CORRECT order
User-agent: GPTBot
Allow: /blog/
Disallow: /

How to Test Your robots.txt

Google Search Console Robots.txt Tester

Google Search Console (search.google.com/search-console) includes a built-in tester. Submit any URL on your domain and it will tell you whether Googlebot can or cannot access it based on your current robots.txt. While it only tests Googlebot and Bingbot natively, it is the fastest way to catch syntax errors.

Fetch and Inspect Manually

curl -s https://yourdomain.com/robots.txt

Confirm the file returns HTTP 200 (not 301, 302, or 404). A redirect on robots.txt can confuse crawlers.

Online Robots.txt Validators

  • robots-checker.io, validates syntax and simulates bot access
  • technicalseo.com/tools/robots-txt-tester, per-bot simulation
  • AY Rank Robots.txt Generator, /tools/robots-txt-generator, generate and validate AI-optimized configurations

Check Actual Bot Behaviour

In your web server access logs, filter for AI bot user-agent strings:

grep -i "GPTBot\|OAI-SearchBot\|ClaudeBot\|PerplexityBot\|Google-Extended" /var/log/nginx/access.log

If blocked bots continue to appear after you set Disallow: /, they are either: (a) not complying with the protocol (common with some scrapers), or (b) still processing a cached version of your robots.txt (wait 24-48 hours for re-crawl).


robots.txt and llms.txt: Using Both Together

The emerging llms.txt standard (proposed by Answer.AI) is a companion file, not a replacement for robots.txt. While robots.txt controls crawler access, llms.txt provides curated guidance, a structured summary of your most important content for LLMs to use.

A complete AI-visibility setup uses both:

  • robots.txt, access control layer (allow/block crawlers)
  • llms.txt, curation layer (direct LLMs to your best content)
  • Schema.org structured data, context layer (help LLMs understand what your content is about)

For the full checklist, including server log monitoring and JS-rendering fixes, see our guide on how to make your website readable by AI agents.


robots.txt for Common CMS Platforms

Next.js App Router

Next.js 13+ supports a robots.ts file at src/app/robots.ts:

import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/' },
      { userAgent: 'GPTBot', disallow: '/' },
      { userAgent: 'ClaudeBot', disallow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },
      { userAgent: 'OAI-SearchBot', allow: '/' },
      { userAgent: 'PerplexityBot', allow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

WordPress

WordPress generates a virtual robots.txt. Use a plugin like Yoast SEO or Rank Math to edit it, or add a physical robots.txt file to your server root (which overrides the virtual one).

Shopify

Shopify does not allow direct editing of robots.txt in standard plans. Use the Liquid-based robots.txt.liquid template (available in Online Store > Themes > Edit Code) to customise directives.


Monitoring and Updating Your robots.txt

AI crawler user-agent strings change as platforms launch new products. Build a quarterly review into your SEO workflow:

  1. Check operator documentation, OpenAI, Anthropic, Google, and Perplexity all publish their current bot names
  2. Review your access logs for new bot strings you do not recognise
  3. Monitor AI referral traffic in GA4, unexpected drops may indicate an accidental block
  4. Re-validate after CMS updates, platform updates sometimes regenerate or overwrite robots.txt

A correctly configured robots.txt only gets a crawler to your page. It does not make client-rendered widgets (booking calendars, pricing tables, FAQ accordions) readable once it arrives; see why a JS-only booking widget is invisible to AI search for that separate, and just as common, gap.

If your site sits behind Cloudflare, read what changed with the Cloudflare AI crawler block.

Frequently Asked Questions

No. GPTBot collects data to train OpenAI's models. Appearing in ChatGPT search results is controlled separately by OAI-SearchBot, which crawls for real-time retrieval, and ChatGPT-User, which fires during live browsing. You can block GPTBot to opt out of training and still allow the other two, so ChatGPT can keep surfacing and citing your pages.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended only controls whether your content trains Gemini and future Google AI models. AI Overviews are driven by standard Googlebot indexing, and robots.txt cannot control them separately. Applebot-Extended works the same way for Apple: blocking it limits Apple Intelligence training without affecting Siri search visibility.

Which AI crawlers should I allow for GEO?

The most important bots for GEO visibility are OAI-SearchBot, PerplexityBot and Googlebot. PerplexityBot matters because Perplexity is a search-first product, and blocking its crawler removes your site from Perplexity results entirely. For maximum AI visibility, allow all search-oriented bots and block only pure training scrapers if training use concerns you.

Do I need to block both ClaudeBot and anthropic-ai?

Yes. Anthropic uses two user-agent strings: ClaudeBot, which trains Claude models, and anthropic-ai, a legacy string from the same operator. A robots.txt that mentions only one will not fully restrict the other, so include both in any allow or block directive you write for Anthropic.

Can robots.txt stop AI companies from using my content?

Only partly. robots.txt works on the honour system, so compliant crawlers respect it and determined bad actors can ignore it. It cannot remove content from models that already ingested it or pull cached content out of existing training sets, and it is no substitute for server-side authentication on sensitive content. Some researchers have reported Bytespider ignoring robots.txt directives.

How do I check whether AI bots respect my robots.txt?

Fetch the file with curl and confirm it returns HTTP 200, not a redirect or a 404. Then filter your server access logs for AI user-agent strings such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. If a blocked bot keeps appearing, it is either ignoring the protocol or still using a cached copy of your robots.txt, so allow 24 to 48 hours for a re-crawl.

Key Takeaways

  • AI crawlers fall into two categories: search-retrieval bots (drive traffic, allow these) and training scrapers (ingest data for model training, allow or block based on your content strategy)
  • The most important bots for GEO visibility are OAI-SearchBot, PerplexityBot, and Googlebot
  • Block training scrapers with targeted Disallow: / rules under specific User-agent: blocks, not with a wildcard
  • Always include Applebot-Extended and Google-Extended in your strategy, they are training-only bots that are completely separate from search indexing
  • Test your configuration with Google Search Console and manual log analysis
  • Use our Robots.txt Generator to build a validated, AI-optimised configuration in under two minutes

This post is part of our Technical SEO guide. Related reading: How to Optimize for Google AI Overviews, E-E-A-T for AI Search, Complete Guide to llms.txt.

About the Author
Abdelmoghit Idhsaine
Abdelmoghit Idhsaine
Content Strategist

Abdelmoghit drives the content engine at AY Rank. He researches keywords, plans content clusters, and produces citation-optimized articles that rank in both Google and AI search engines.

Full Bio →
More From the Blog
How We Built the Citation Readiness Score

How We Built the Citation Readiness Score

The full methodology behind our AI Citation Predictor: five factors, every point value, the research each leans on, and what the score cannot tell you.

Read article →
Cloudflare's September 15 AI Crawler Change: What to Check

Cloudflare's September 15 AI Crawler Change: What to Check

Cloudflare's new AI crawler defaults took effect September 15, 2026. Here's what changed, who it hit, and the self-block risk worth checking now.

Read article →
Artificial Analysis Index: What It Means for GEO

Artificial Analysis Index: What It Means for GEO

Three frontier models shipped within three days in September 2026. The independent Artificial Analysis Intelligence Index first called Astra and Fable 5.1 a tie, then rescored them under v5. Here is what moving scores mean if you track AI visibility for a living.

Read article →
AY Rank GEO strategistAY Rank SEO specialistAY Rank client success lead
Dominate AI SearchOthers are ranking. Start before they finish.
»