Blog
4 August 2026/12 min read

robots.txt for AI Crawlers: Complete Setup Guide

A complete guide to configuring robots.txt for every major AI crawler, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more. Learn which bots to allow, which to block, and how to test your configuration.

Run a free check on app.ayrank.comNo sign-up. See if ChatGPT names your brand.WearviewClient result: Wearview went from $0 to $36K MRR in 8 months.
Abdelmoghit Idhsaine
Author:Abdelmoghit Idhsaine,Content Strategist
robots.txt for AI Crawlers: Complete Setup Guide

This post is part of our Technical SEO guide. Related reading: How to Optimize for Google AI Overviews, E-E-A-T for AI Search, Complete Guide to llms.txt.

About the Author
Abdelmoghit Idhsaine
Abdelmoghit Idhsaine
Content Strategist

Abdelmoghit drives the content engine at AY Rank. He researches keywords, plans content clusters, and produces citation-optimized articles that rank in both Google and AI search engines.

Full Bio →
More From the Blog
Brands are blocking AI agents: what United's move means for agent visibility

Brands are blocking AI agents: what United's move means for agent visibility

On November 11, 2026, United said it will restrict unauthorized third-party AI agents across its platforms and through agencies. Here is exactly what United said, what it left out, and why agents acting in an account are a different problem from the crawlers that put you in AI answers.

Read article →
CDN migration SEO checklist: what Google says to do before, during and after

CDN migration SEO checklist: what Google says to do before, during and after

Google's November 11, 2026 post explains why crawling hits your origin harder right after a CDN move, and how blanket bot rules can push pages out of Search. A before, during and after checklist built on Google's own words.

Read article →
How to read Bing's AI Topics Portfolio and turn gaps into pages

How to read Bing's AI Topics Portfolio and turn gaps into pages

Bing Webmaster Tools added Topics Portfolio (Beta) to its AI Performance report on October 13, 2026. Where it is, what each column and range means in Bing's words, and a checklist that turns each opportunity query into a page.

Read article →
AY Rank GEO strategistAY Rank SEO specialistAY Rank client success lead
Get your free AI auditOthers are ranking. Start before they finish.
»