robots.txt for AI Crawlers: Complete Setup Guide

robots.txt for AI Crawlers: Complete Setup Guide

Your robots.txt file is no longer just a directive for Googlebot and Bingbot. In 2026 it has become the primary access-control layer for a new generation of AI crawlers, systems that feed ChatGPT, Claude, Perplexity, Google AI Overviews, Apple Intelligence, and dozens of other large-language-model products.

Getting this file wrong has real business consequences: you may accidentally block crawlers that drive referral traffic, or leave your content exposed to scrapers that train competing AI models without compensating you.

This guide covers every major AI crawler, explains what each one does, and gives you a production-ready robots.txt template you can deploy today.

---

What robots.txt Actually Does for AI Crawlers

The Robots Exclusion Protocol is a plain-text convention, not a legal contract or enforced standard. It works on the honour system: you publish directives, and compliant crawlers respect them. The key word is compliant.

What robots.txt can do:

• Signal to well-behaved AI crawlers which pages or paths to avoid
• Distinguish between indexing bots and training-data scrapers
• Grant or deny access on a per-bot-name basis
• Set crawl-delay hints to protect server resources

What robots.txt cannot do:

• Block determined bad actors who ignore the protocol
• Prevent your already-cached content from being used in existing AI training sets
• Remove content from AI models that already ingested it
• Substitute for robust server-side authentication on sensitive content

With those boundaries clear, a well-crafted robots.txt is still your first line of defence and the easiest lever you have.

---

The Complete List of AI Crawlers (2026)

The AI crawler landscape has grown rapidly. Here is every significant bot you need to know about, organized by operator.

OpenAI

GPTBot is the big one. If you block it, your content will not be used to improve future OpenAI models. However, blocking GPTBot does not prevent your site from appearing in ChatGPT search results, that is controlled separately by OAI-SearchBot and ChatGPT-User.

OAI-SearchBot crawls for real-time retrieval. Blocking it means ChatGPT cannot surface your pages when users ask questions that would otherwise cite you. For most content marketers this is traffic you want.

ChatGPT-User fires per user session during live browsing. It typically respects the same rules as OAI-SearchBot.

Anthropic (Claude)

Anthropic uses two different user-agent strings. To be safe, include both in any allow or block directive.

Perplexity AI

Perplexity is a search-first product, it surfaces citations to users actively searching. Blocking PerplexityBot removes your site from Perplexity results entirely. For GEO-focused strategies, this bot is one of the highest-priority crawlers to allow.

Google

Google-Extended was introduced specifically to give publishers control over AI training separately from search indexing. Blocking Google-Extended will not remove you from AI Overviews, that is driven by standard Googlebot indexing and is not separately controllable via robots.txt.

Apple

Same logic as Google: blocking Applebot-Extended limits AI training without affecting Siri search visibility.

Meta

Facebook link previews are driven by FacebookBot. If you block it entirely, your URLs will show blank previews when shared on Facebook and Instagram.

Cohere

Cohere primarily serves B2B enterprise AI products. Less likely to drive direct referral traffic.

ByteDance / TikTok

Bytespider has a mixed reputation for protocol compliance. Some security researchers have reported it ignoring robots.txt directives. Blocking it is a reasonable precaution if you are concerned about scraping.

Other Notable Bots

---

How to Allow or Block Each Crawler

The syntax is straightforward. Each block begins with a User-agent: line followed by Allow: or Disallow: directives.

Block a specific bot entirely

User-agent: GPTBot
Disallow: /

Allow a specific bot to access everything

User-agent: PerplexityBot
Allow: /

Block a bot from specific paths only

User-agent: GPTBot
Disallow: /private/
Disallow: /members/
Allow: /blog/
Allow: /

Allow a bot despite a broader Disallow rule

User-agent: *
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Important: Bot-specific blocks take precedence over the wildcard * block. Declare specific agents after the wildcard block, not before.

---

Recommended Strategy by Goal

Before writing your robots.txt, decide what you actually want from AI platforms:

Goal A: Maximum AI search visibility (GEO-first strategy) Allow all search-oriented bots. Block only pure training scrapers if you have concerns about AI training use.

Goal B: Block AI training, preserve search Allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot. Block GPTBot, ClaudeBot, Google-Extended, anthropic-ai, cohere-ai, Bytespider.

Goal C: Block all AI activity Block every AI bot. This is rarely advisable for marketing sites as it removes you from an increasingly significant traffic source.

Goal D: Selective by content type Allow bots on public marketing content, block on proprietary research or gated resources.

---

Complete Production robots.txt Template

Here is a fully-annotated template. Remove the comment lines before deploying.

# =============================================================
# robots.txt for [yourdomain.com]
# Last updated: 2026-05-12
# Strategy: Allow AI search bots, block AI training scrapers
# =============================================================

# --- Standard search engines (always allow) ---
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Slurp
Allow: /

User-agent: DuckDuckBot
Allow: /

# --- OpenAI: allow search, block training ---
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

# --- Anthropic: allow search, block training ---
# (Claude does not yet have a separate search bot)
User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

# --- Perplexity: allow (search-first product) ---
User-agent: PerplexityBot
Allow: /

# --- Google AI training (separate from Googlebot) ---
User-agent: Google-Extended
Disallow: /

# --- Apple: allow Siri search, block AI training ---
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

# --- Meta: allow link previews, review training ---
User-agent: FacebookBot
Allow: /

User-agent: meta-externalagent
Disallow: /

# --- Cohere: block training ---
User-agent: cohere-ai
Disallow: /

# --- ByteDance / TikTok ---
User-agent: Bytespider
Disallow: /

# --- Common Crawl (used for open LLM training) ---
User-agent: CCBot
Disallow: /

# --- You.com AI search ---
User-agent: YouBot
Allow: /

# --- Allen Institute ---
User-agent: AI2Bot
Disallow: /

# --- Default: allow everything not covered above ---
User-agent: *
Allow: /

# --- Sitemap location (always include) ---
Sitemap: https://yourdomain.com/sitemap.xml

Template for maximum AI search visibility (GEO-first)

If your goal is citation and referral traffic from every AI platform, use this simpler version:

# Allow all AI crawlers for maximum GEO visibility
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

---

Common robots.txt Mistakes to Avoid

1. Using a wildcard block without exceptions

# WRONG, blocks Googlebot too
User-agent: *
Disallow: /

If you use a wildcard Disallow, you must explicitly allow each bot you want to permit. Many site owners inadvertently block Googlebot this way.

2. Confusing training bots with search bots

Blocking GPTBot (training) is different from blocking OAI-SearchBot (search retrieval). Blocking the wrong one achieves neither your training-opt-out goal nor your GEO visibility goal.

3. Forgetting anthropic-ai alongside ClaudeBot

Anthropic uses both ClaudeBot and anthropic-ai as user-agent strings. A robots.txt that only mentions one will not fully restrict the other.

4. Placing specific bot rules before the wildcard

Some parsers handle ordering differently. Best practice is: wildcard rule first, then specific bot rules. Specific rules always override the wildcard regardless of order, but this convention makes your file easier to audit.

5. Not including a Sitemap directive

The Sitemap: line at the bottom of robots.txt is how AI crawlers discover your content index. Omitting it is a missed opportunity.

6. Blocking /api/ paths that serve public content

Some Next.js and headless CMS sites serve rendered content via API routes. If AI crawlers are blocked from those paths, they may not access the content even if the marketing pages are allowed.

7. Inconsistent www vs. non-www

rob.txt at www.domain.com/robots.txt applies only to the www subdomain. domain.com/robots.txt covers the apex domain. Most sites should have matching content at both.

8. Declaring Disallow before Allow in the same block

# WRONG order
User-agent: GPTBot
Disallow: /
Allow: /blog/

In many parsers, Disallow: / catches everything before Allow: /blog/ is processed. Use the most specific rule first:

# CORRECT order
User-agent: GPTBot
Allow: /blog/
Disallow: /

---

How to Test Your robots.txt

Google Search Console Robots.txt Tester

Google Search Console (search.google.com/search-console) includes a built-in tester. Submit any URL on your domain and it will tell you whether Googlebot can or cannot access it based on your current robots.txt. While it only tests Googlebot and Bingbot natively, it is the fastest way to catch syntax errors.

Fetch and Inspect Manually

curl -s https://yourdomain.com/robots.txt

Confirm the file returns HTTP 200 (not 301, 302, or 404). A redirect on robots.txt can confuse crawlers.

Online Robots.txt Validators

• robots-checker.io, validates syntax and simulates bot access
• technicalseo.com/tools/robots-txt-tester, per-bot simulation
• AY Rank Robots.txt Generator, /tools/robots-txt-generator, generate and validate AI-optimized configurations

Check Actual Bot Behaviour

In your web server access logs, filter for AI bot user-agent strings:

grep -i "GPTBot\|OAI-SearchBot\|ClaudeBot\|PerplexityBot\|Google-Extended" /var/log/nginx/access.log

If blocked bots continue to appear after you set Disallow: /, they are either: (a) not complying with the protocol (common with some scrapers), or (b) still processing a cached version of your robots.txt (wait 24-48 hours for re-crawl).

---

robots.txt and llms.txt: Using Both Together

The emerging llms.txt standard (proposed by Answer.AI) is a companion file, not a replacement for robots.txt. While robots.txt controls crawler access, llms.txt provides curated guidance, a structured summary of your most important content for LLMs to use.

A complete AI-visibility setup uses both:

• robots.txt, access control layer (allow/block crawlers)
• llms.txt, curation layer (direct LLMs to your best content)
• Schema.org structured data, context layer (help LLMs understand what your content is about)

For the full checklist, including server log monitoring and JS-rendering fixes, see our guide on how to make your website readable by AI agents.

---

robots.txt for Common CMS Platforms

Next.js App Router

Next.js 13+ supports a robots.ts file at src/app/robots.ts:

import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/' },
      { userAgent: 'GPTBot', disallow: '/' },
      { userAgent: 'ClaudeBot', disallow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },
      { userAgent: 'OAI-SearchBot', allow: '/' },
      { userAgent: 'PerplexityBot', allow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

WordPress

WordPress generates a virtual robots.txt. Use a plugin like Yoast SEO or Rank Math to edit it, or add a physical robots.txt file to your server root (which overrides the virtual one).

Shopify

Shopify does not allow direct editing of robots.txt in standard plans. Use the Liquid-based robots.txt.liquid template (available in Online Store > Themes > Edit Code) to customise directives.

---

Monitoring and Updating Your robots.txt

AI crawler user-agent strings change as platforms launch new products. Build a quarterly review into your SEO workflow:

1. Check operator documentation, OpenAI, Anthropic, Google, and Perplexity all publish their current bot names
2. Review your access logs for new bot strings you do not recognise
3. Monitor AI referral traffic in GA4, unexpected drops may indicate an accidental block
4. Re-validate after CMS updates, platform updates sometimes regenerate or overwrite robots.txt

---

A correctly configured robots.txt only gets a crawler to your page. It does not make client-rendered widgets (booking calendars, pricing tables, FAQ accordions) readable once it arrives; see why a JS-only booking widget is invisible to AI search for that separate, and just as common, gap.

Key Takeaways

• AI crawlers fall into two categories: search-retrieval bots (drive traffic, allow these) and training scrapers (ingest data for model training, allow or block based on your content strategy)
• The most important bots for GEO visibility are OAI-SearchBot, PerplexityBot, and Googlebot
• Block training scrapers with targeted Disallow: / rules under specific User-agent: blocks, not with a wildcard
• Always include Applebot-Extended and Google-Extended in your strategy, they are training-only bots that are completely separate from search indexing
• Test your configuration with Google Search Console and manual log analysis
• Use our Robots.txt Generator to build a validated, AI-optimised configuration in under two minutes