To make your website readable by AI agents, do five things: publish an llms.txt file that maps your content, monitor AI crawler visits (GPTBot, ClaudeBot, PerplexityBot) in your server logs, serve a clean markdown version of every page, honour content negotiation so agents asking for text/markdown get it, and replace JS-only widgets with server-rendered fallbacks. Most sites do at most one of the five. We know because we audited our own site in August 2026 and scored one out of five. This guide is the exact fix list we built for ourselves, step by step.
Why AI-agent readability is now its own discipline
AI agents read the web differently from browsers. Training crawlers fetch your pages to build model knowledge; retrieval bots fetch them live when a user asks ChatGPT, Perplexity, or Claude a question your page could answer. Neither executes your JavaScript the way a browser does, neither cares about your hero animation, and both reward sites that hand over clean, structured text. If your content is locked behind client-side rendering, an AI agent sees an empty page, and an empty page never gets cited.
What an AI agent actually sees on your site: the agent arrives, robots.txt and llms.txt open the door, clean markdown makes the content extractable, and JS-only widgets render as an empty div that does not exist for most agents
Know the crawlers you are optimising for
AI crawlers split into two families, and the split matters because you may want different policies for each (reference lists; full 2026 landscape):
| Family | User agents | What they do |
|---|
| Training crawlers | GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, Applebot-Extended, Bytespider, CCBot | Fetch public content for model training. Blocking them keeps you out of future model knowledge. |
| Retrieval and search bots | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User | Fetch pages live to ground answers and citations. Blocking them removes you from AI answers today. |
For GEO, the retrieval family is the one that pays: OAI-SearchBot and Claude-SearchBot are the citation pipelines of ChatGPT search and Claude respectively. The major providers state their training crawlers respect robots.txt, so the control is real, in both directions.
Step 1: publish an llms.txt
llms.txt is a markdown file at your site root that gives AI agents a curated map of what matters: your key pages, what each one covers, and where the canonical answers live. It is the one item our own audit passed, and the reason is instructive: ours is generated from the site's data sources on every content change, never hand-edited, so it cannot drift out of date. A stale llms.txt is worse than none, because it hands agents dead links with your name on them.
Keep it curated rather than exhaustive: lead with the pages you want quoted, one line of description each. If your content runs deep, add an llms-full.txt with the expanded page list.
Step 2: monitor AI crawlers in your server logs
You cannot manage what you cannot see, and AI crawler visits are invisible in normal analytics because bots do not fire your JavaScript trackers. The fix is a few lines in your middleware or edge function: match the user-agent header against the crawler names above and log a structured line (agent, path, timestamp).
Our audit failed this check, and the fix we built is deliberately small: a regex over GPTBot|PerplexityBot|ClaudeBot|Google-Extended|CCBot|Bytespider|Amazonbot in the middleware that already runs on every request, logging to the platform's function logs. No blocking, no latency, and within a week you know which agents visit, which pages they want, and whether a robots.txt change actually changed behaviour. If your log lines include the client IP, our AI crawler log checker shows which GPTBot, ClaudeBot or Googlebot hits came from the vendor's own IP ranges.
Step 3: serve markdown per page
HTML built for browsers is noisy for agents: navigation, cookie banners, and layout markup dilute the text they came for. The emerging convention is to serve a clean markdown twin of each content page, either at a parallel URL (/blog/your-post.md) or via content negotiation (step 4).
On our own audit, /blog/llm-seo.md returned a 404: fail. The fix on a modern stack is one route handler that reuses the same cached content fetch the HTML page uses and returns it as Content-Type: text/markdown. No duplicate content risk follows from this: it is the same canonical content in a second representation, exactly like a print stylesheet was in 2005.
Step 4: honour content negotiation
Sophisticated agents do not guess URLs; they ask. A request with Accept: text/markdown is an agent saying "give me the clean version if you have one". Most sites ignore the header and return HTML. Honouring it takes roughly fifteen lines of middleware: when the Accept header prefers text/markdown or text/plain on a content URL, route the request to the markdown twin from step 3. The step past markdown is letting agents call your site directly, which we did with WebMCP on our own site.
Our audit failed this one too, and the pattern of the fix matters more than the code: markdown route first, negotiation as a thin layer on top. Do not build negotiation without the twin, and do not redirect agents to a PDF.