AI SEO AgencyServicesCase StudiesPlansToolsBlogOur WorkflowsWhat is GEOHow it works
Free AI Audit
Free Tool

See which AI crawlers really visited your site

Paste server access log lines or open a log file. See hits per AI crawler and whether each one came from the vendor's published IP ranges. Your log never leaves your browser.

Your access log

Your log stays in this tab. The page reads what you paste or open with JavaScript on your device and matches every address here. It downloads the vendors' IP lists once and uploads nothing.

Reads the first ⁦50⁩ MB or ⁦500,000⁩ lines. A .log, .txt or JSON lines file is read in your browser. Unzip .gz files first.

Loading the vendors' IP lists...

robots.txt says who may crawl. Logs say who came.

Test whether GPTBot or any other crawler may fetch a URL under your robots.txt rules.

Open the robots.txt tester

Next step: check AI bot access for any domain

The AI crawler checker in the AY Rank app reads a domain's robots.txt for GPTBot, ClaudeBot and 10 more AI bots.

Open the AI crawler checker
1

Paste or open a log

Paste access log lines or pick a log file. Your browser reads the file. Nothing is uploaded.

2

Find the AI crawlers

Each line's client address and user agent are read, and 17 crawler tokens such as GPTBot, ClaudeBot and PerplexityBot are looked for.

3

Match the address

Each address is compared with the IPv4 and IPv6 ranges the vendor publishes, on your device.

4

Read the verdicts

Verified, likely spoofed or unverifiable per crawler, with top paths and status codes for verified hits.

robots.txt says who may crawl. Logs say who came. A user agent string is free to copy, so a GPTBot line in your log proves nothing until its address is in OpenAI's list. The address check tells you which AI crawlers really read your pages, which pages they read and what status code they got.

How each verdict is decided

  1. Verified: the user agent names a crawler and the address is inside one of the ranges the vendor publishes for that crawler.
  2. Likely spoofed: the user agent names a crawler but the address is outside the vendor's ranges. Anyone can copy a user agent, so this is the usual sign of a fake bot. It is not proof: a vendor can start using a new address before its list is updated.
  3. Unverifiable: the vendor publishes no IP list, the list has no ranges for that IP version (Bing's list has IPv4 ranges only), the line has no address, or the list could not be loaded. The tool never calls these hits fake.
  4. The tool reads the first address on the line, the one your server logged as the client, and leaves X-Forwarded-For as it is. Behind a CDN, proxy or load balancer that address can be the proxy, so set your server to log the real client IP (for example the Nginx realip module or Apache mod_remoteip) before you trust the verdicts.
  5. User agent tokens match whole words and ignore case. ChatGPT-User is not GPTBot, Claude-User is not ClaudeBot, and Googlebot-Image or Applebot-Extended do not count as Googlebot or Applebot.
  6. Formats read: Apache and Nginx combined and common log formats, W3C extended logs with a #Fields line (IIS), JSON lines (Cloudflare Logpush fields like ClientIP and ClientRequestUserAgent, and keys like remote_addr and http_user_agent), and other lines where the tool finds an IPv4 or IPv6 address and a quoted user agent. The common log format has no user agent, so those lines cannot name a crawler.
  7. Limits: the first 50 MB or 500,000 lines. Compressed .gz files must be unzipped first. Top paths drop the query string.
  8. The IP lists come from /api/v1/ai-crawler-ip-ranges, which fetches only the vendor files listed below, at most once a day. If a vendor file cannot be read, it serves the copy saved on 2026-11-11 and marks it as a saved copy.

Crawlers and their IP lists

The 17 crawlers this tool checks, with the user agent token and the IP list each vendor publishes. Read on the vendors' own pages on 2026-11-11.

  • Anthropic publishes one list for ClaudeBot, Claude-SearchBot and Claude-User. Its help page says: "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic."
  • Google's common-crawlers.json covers Googlebot and Google's other common crawlers. Google also documents a reverse DNS check: the host name must end in googlebot.com, google.com or googleusercontent.com.
  • Bing's bingbot.json has IPv4 ranges only, so a Bingbot hit over IPv6 shows as unverifiable. Bing's help article on verifying Bingbot describes a reverse DNS and forward lookup check for those.
  • Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. Google says Google-Extended "doesn't have a separate HTTP request user agent string" and Apple says "Applebot-Extended does not crawl webpages." Their fetches show up as Googlebot and Applebot.

Sources

Each page and IP list read on 2026-11-11.

Frequently Asked Questions

Is my log uploaded anywhere?

No. The page reads your text or file with JavaScript in your browser and matches addresses there. Its only request is one GET to /api/v1/ai-crawler-ip-ranges for the vendors' IP lists, and that request carries no log data. Close the tab and the log is gone.

How do I verify GPTBot by IP address?

OpenAI publishes GPTBot's addresses at openai.com/gptbot.json, with separate files for OAI-SearchBot, ChatGPT-User and OAI-AdsBot. A GPTBot hit is real when its address falls inside one of the ranges in gptbot.json. This tool runs that check on every line.

Why does a real crawler show as likely spoofed?

Most often your server logs a CDN, proxy or load balancer address instead of the visitor's. Fix the logging so the client IP comes first. A vendor can also start using a new address before its list catches up, so check again with a fresh list before you block anything.

Why are Meta's crawlers unverifiable?

Meta's crawler page lists the meta-externalagent and meta-externalfetcher user agents but no IP list. Without a list the tool cannot tell a real hit from a copied user agent, so it says unverifiable and never calls the hit fake.

What about Google-Extended and Applebot-Extended?

They are robots.txt tokens, not crawlers. Google says Google-Extended has no separate user agent string, and Apple says Applebot-Extended does not crawl webpages. Their fetches show up as Googlebot and Applebot, so check those rows.

Which log formats work?

Apache and Nginx combined and common formats, IIS and other W3C logs with a #Fields line, JSON lines such as Cloudflare Logpush, and most other formats that put the client address and a quoted user agent on one line. Up to 50 MB or 500,000 lines.

Want us to fix this for you?

We read your server logs, fix what blocks real AI crawlers and work on getting your pages cited in ChatGPT, Perplexity and Google AI answers.

Explore GEO OptimizationBook a free audit»