AI crawler log analysis: who actually reads your site

How to analyze server logs for AI crawlers: filter the tokens, verify the bots, count status codes and see which pages they read.

·5 min read

AI crawler log analysis is the process of filtering your server or CDN logs for the user agents AI companies use, verifying that those requests are genuine, and counting what the crawlers fetched. It answers three questions no dashboard answers: can AI crawlers reach the site, which pages do they read, and is a firewall or robots rule blocking them.

The user agent alone is not proof. Anyone can send a request that claims to be GPTBot. Verification is the part most log reports skip.

Which tokens to look for

Crawler Operator Purpose User agent token
GPTBot OpenAI Model training GPTBot
OAI-SearchBot OpenAI Search and citations OAI-SearchBot
ChatGPT-User OpenAI User-initiated fetch ChatGPT-User
ClaudeBot Anthropic Model training ClaudeBot
Claude-SearchBot Anthropic Search and citations Claude-SearchBot
Claude-User Anthropic User-initiated fetch Claude-User
PerplexityBot Perplexity Search index PerplexityBot
Perplexity-User Perplexity User-initiated fetch Perplexity-User
Bytespider ByteDance Crawling and training Bytespider
CCBot Common Crawl Open corpus CCBot
Googlebot Google Search Googlebot
Bingbot Microsoft Search Bingbot

Two tokens do not have their own user agent: Google-Extended and Applebot-Extended are robots.txt controls that tell Google and Apple whether their content may be used for AI. You will not find a request from them in the logs. Their effect shows up as a change in what Googlebot or Applebot fetches, if at all.

Filter the logs

A quick grep is enough to see whether AI crawlers arrive at all.

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|CCBot" access.log \
  | awk '{print $1, $7, $9}' \
  | sort | uniq -c | sort -rn | head -40

That prints the IP, path and status for each match, most frequent first. Repeat with Googlebot and Bingbot to compare AI crawlers with search crawlers.

For a CDN such as Cloudflare, export the request logs or query the analytics for bot traffic. The field names differ, but the filter is the same.

Verify the bot before you count it

A spoofed request is worse than no request, because it makes a blocked site look crawled.

  • Reverse DNS. Resolve the IP to a hostname, then resolve the hostname back to the IP. Google publishes the googlebot.com and google.com domains; Microsoft publishes search.msn.com. A forward-confirmed reverse DNS lookup is the standard check.
  • Published IP ranges. OpenAI publishes its crawler IP ranges, and Google publishes its own in a JSON file. If the IP is outside the range, the user agent is a lie.
  • Cloudflare verified bots. If the site is behind Cloudflare, its verified bot check does the work for you.
  • Behavior. A real crawler requests HTML and assets at a steady rate. A spoofed one often hits the same URL repeatedly or probes admin paths.

An unverified request still matters for security, but it does not tell you whether OpenAI can read the page. Expect three outcomes, not two: verified, spoofed, and unverifiable when the operator publishes no address range to check against. The last one is not a failure, just missing evidence.

Run the analysis on a pasted log

You do not need to install anything to get the first numbers. Take an export of the log, cut it to the window you care about, and paste it into the analyzer.

  1. Export the access log for the period, or copy the last few thousand lines from the server.
  2. Keep the full lines. Verification needs the connecting IP, the user agent and the status, so trimming columns breaks the report.
  3. Paste the lines into the tool. It accepts combined, JSON, W3C extended and generic line formats, up to 750,000 characters per run.
  4. Read the verdict first, then the per-crawler table. The verdict names the problem; the table names the crawler and the page.
  5. Repeat monthly with the same window so the comparison means something.

A single day is too short. Crawlers arrive on their own schedule, so compare month to month, or week to week on a site that publishes often.

What to measure

Metric Question it answers
Requests per crawler Is the bot visiting at all?
Verified vs spoofed Is the traffic real?
Status codes Is the bot getting 200, 403, 429 or 5xx?
Top paths Which pages does the bot care about?
Bytes served How much bandwidth does the crawler use?
Last seen Is a crawler that used to visit now gone?

The status codes are the most actionable. A crawler that receives 403 on every request is blocked by a firewall, even if robots.txt says allow. A crawler that receives 429 is being rate-limited. A crawler that receives 200 but only on the homepage may be following no internal links.

Read the result like a diagnosis

Pattern Likely cause Fix
No AI crawler requests robots.txt, firewall or no discovery Check the robots.txt group and the WAF
Only spoofed requests Real crawlers blocked before the origin Check the CDN and bot management rules
403 on article pages A bot rule or login wall Allow the crawler token on public content
429 across the site Rate limiting too aggressive for the crawl rate Raise the limit or slow the crawler
Requests to the homepage only Weak internal links or a JS-rendered nav Add server-rendered links
Requests to old URLs Stale sitemap or redirects Update the sitemap

Do not judge AI readiness from a single day. Crawlers visit on their own schedule, and a training crawler may fetch a page once a month.

Logs are necessary, not sufficient

A clean log tells you the crawler can reach the page. It does not tell you whether the page was used in an answer. Pair the log with a citation check: sample the questions your buyers ask and record which domains the AI answer cites.

Our AI bot log analyzer takes a pasted log, matches every known AI and search crawler token in it, checks each request against the operator's published address ranges, and reports hits, verified, spoofed and unverifiable counts, blocked and error responses, the top paths each crawler reads, and the last time it appeared. Start there, then read log file analysis for the wider crawl picture, the AI crawler list for the tokens, and how to block AI crawlers if the goal is the opposite.

Related guidesFrom the same topic, then the newest guides.