How to make your website readable by AI agents (GPTBot, ClaudeBot, PerplexityBot)

To make your website readable by AI agents, do five things: publish an llms.txt file that maps your content, monitor AI crawler visits (GPTBot, ClaudeBot, PerplexityBot) in your server logs, serve a clean markdown version of every page, honour content negotiation so agents asking for text/markdown get it, and replace JS-only widgets with server-rendered fallbacks. Most sites do at most one of the five. We know because we audited our own site in August 2026 and scored one out of five. This guide is the exact fix list we built for ourselves, step by step.

Why AI-agent readability is now its own discipline

AI agents read the web differently from browsers. Training crawlers fetch your pages to build model knowledge; retrieval bots fetch them live when a user asks ChatGPT, Perplexity, or Claude a question your page could answer. Neither executes your JavaScript the way a browser does, neither cares about your hero animation, and both reward sites that hand over clean, structured text. If your content is locked behind client-side rendering, an AI agent sees an empty page, and an empty page never gets cited.

What an AI agent actually sees on your site: the agent arrives, robots.txt and llms.txt open the door, clean markdown makes the content extractable, and JS-only widgets render as an empty div that does not exist for most agents

Know the crawlers you are optimising for

AI crawlers split into two families, and the split matters because you may want different policies for each (reference lists; full 2026 landscape):

For GEO, the retrieval family is the one that pays: OAI-SearchBot and Claude-SearchBot are the citation pipelines of ChatGPT search and Claude respectively. The major providers state their training crawlers respect robots.txt, so the control is real, in both directions.

Step 1: publish an llms.txt

llms.txt is a markdown file at your site root that gives AI agents a curated map of what matters: your key pages, what each one covers, and where the canonical answers live. It is the one item our own audit passed, and the reason is instructive: ours is generated from the site's data sources on every content change, never hand-edited, so it cannot drift out of date. A stale llms.txt is worse than none, because it hands agents dead links with your name on them.

Keep it curated rather than exhaustive: lead with the pages you want quoted, one line of description each. If your content runs deep, add an llms-full.txt with the expanded page list.

Step 2: monitor AI crawlers in your server logs

You cannot manage what you cannot see, and AI crawler visits are invisible in normal analytics because bots do not fire your JavaScript trackers. The fix is a few lines in your middleware or edge function: match the user-agent header against the crawler names above and log a structured line (agent, path, timestamp).

Our audit failed this check, and the fix we built is deliberately small: a regex over GPTBot|PerplexityBot|ClaudeBot|Google-Extended|CCBot|Bytespider|Amazonbot in the middleware that already runs on every request, logging to the platform's function logs. No blocking, no latency, and within a week you know which agents visit, which pages they want, and whether a robots.txt change actually changed behaviour.

Step 3: serve markdown per page

HTML built for browsers is noisy for agents: navigation, cookie banners, and layout markup dilute the text they came for. The emerging convention is to serve a clean markdown twin of each content page, either at a parallel URL (/blog/your-post.md) or via content negotiation (step 4).

On our own audit, /blog/llm-seo.md returned a 404: fail. The fix on a modern stack is one route handler that reuses the same cached content fetch the HTML page uses and returns it as Content-Type: text/markdown. No duplicate content risk follows from this: it is the same canonical content in a second representation, exactly like a print stylesheet was in 2005.

Step 4: honour content negotiation

Sophisticated agents do not guess URLs; they ask. A request with Accept: text/markdown is an agent saying "give me the clean version if you have one". Most sites ignore the header and return HTML. Honouring it takes roughly fifteen lines of middleware: when the Accept header prefers text/markdown or text/plain on a content URL, route the request to the markdown twin from step 3.

Our audit failed this one too, and the pattern of the fix matters more than the code: markdown route first, negotiation as a thin layer on top. Do not build negotiation without the twin, and do not redirect agents to a PDF.

Step 5: fix JS-only widgets

This is the trap that hits conversion where it hurts. Embedded booking widgets, calculators, and chat launchers are usually a <div> that JavaScript fills in later. A browser user sees your booking calendar; an AI agent sees an empty element. Our own booking embed had exactly this problem: the widget rendered server-side as an empty div, which means the single most important action on the site did not exist for agents reading it.

The fix costs minutes: a <noscript> block (or better, always-rendered adjacent text) containing the real booking URL and the core offer copy. An agent that cannot run your widget can still read, quote, and link the action you want taken.

What our own audit scored, honestly

We are a GEO agency, so we ran this exact five-point checklist against our own site before writing about it:

One out of five, on a site that already ranks and gets cited. That is the honest state of most of the web right now, and it is why the fixes above are ordered by impact: each one we describe is one we built. The same discipline applies to provenance: AI agents reading your site is one half of the new contract, and AI provenance marks like Claude's watermark are the other. Readable in, traceable out.

FAQ

How do I make my website readable by AI agents like GPTBot and ClaudeBot? Five moves cover it: publish a generated llms.txt map, log AI crawler user agents (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) in your server logs, serve a markdown twin of each content page, honour Accept: text/markdown content negotiation, and give every JS-only widget a server-rendered fallback. Allow the retrieval bots in robots.txt or none of the rest matters.

Should I block AI crawlers or allow them? Split the decision by family. Retrieval bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) fetch pages to ground live answers; blocking them removes you from AI citations, which for most commercial sites means blocking your own demand. Training crawlers (GPTBot, ClaudeBot, Google-Extended) are a policy choice about model training; many publishers allow those too, since absence from model knowledge is its own cost.

What is llms.txt and do AI agents actually use it? llms.txt is a markdown file at your site root listing your key pages with one-line descriptions, a curated map for AI agents rather than an exhaustive sitemap. Adoption is growing on the agent side and the cost is near zero, especially if you generate it from your site's data instead of hand-editing. A generated file cannot drift; a hand-edited one always does.

Does serving markdown versions of pages create duplicate content problems? No. A markdown twin is a second representation of the same canonical content at a clearly related URL, not a competing page. Keep the HTML page canonical, serve the markdown with Content-Type: text/markdown, and search engines treat it the way they have always treated alternate representations.

How do I know if AI crawlers are visiting my site? Server logs, not analytics. AI crawlers do not execute JavaScript trackers, so they are invisible to GA4-style tools. Match the user-agent header in your middleware against the known crawler names and log the hits. Within days you will see which agents visit and what they read, which is the baseline every other decision needs.

Do AI agents execute JavaScript? Mostly no, and never reliably. Some retrieval systems render a limited amount, but training crawlers and most agent fetches read the server response as-is. Anything that only exists after client-side JavaScript runs, booking widgets, calculators, tab content, should be assumed invisible and given a server-rendered equivalent.