AI crawler log analysis: who actually reads your site
How to analyze server logs for AI crawlers: filter the tokens, verify the bots, count status codes and see which pages they read.
·5 min read
AI crawler log analysis is the process of filtering your server or CDN logs for the user agents AI companies use, verifying that those requests are genuine, and counting what the crawlers fetched. It answers three questions no dashboard answers: can AI crawlers reach the site, which pages do they read, and is a firewall or robots rule blocking them.
The user agent alone is not proof. Anyone can send a request that claims to be GPTBot. Verification is the part most log reports skip.
Which tokens to look for
| Crawler | Operator | Purpose | User agent token |
|---|---|---|---|
| GPTBot | OpenAI | Model training | GPTBot |
| OAI-SearchBot | OpenAI | Search and citations | OAI-SearchBot |
| ChatGPT-User | OpenAI | User-initiated fetch | ChatGPT-User |
| ClaudeBot | Anthropic | Model training | ClaudeBot |
| Claude-SearchBot | Anthropic | Search and citations | Claude-SearchBot |
| Claude-User | Anthropic | User-initiated fetch | Claude-User |
| PerplexityBot | Perplexity | Search index | PerplexityBot |
| Perplexity-User | Perplexity | User-initiated fetch | Perplexity-User |
| Bytespider | ByteDance | Crawling and training | Bytespider |
| CCBot | Common Crawl | Open corpus | CCBot |
| Googlebot | Search | Googlebot |
|
| Bingbot | Microsoft | Search | Bingbot |
Two tokens do not have their own user agent: Google-Extended and Applebot-Extended are robots.txt controls that tell Google and Apple whether their content may be used for AI. You will not find a request from them in the logs. Their effect shows up as a change in what Googlebot or Applebot fetches, if at all.
Filter the logs
A quick grep is enough to see whether AI crawlers arrive at all.
grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|CCBot" access.log \
| awk '{print $1, $7, $9}' \
| sort | uniq -c | sort -rn | head -40That prints the IP, path and status for each match, most frequent first. Repeat with Googlebot and Bingbot to compare AI crawlers with search crawlers.
For a CDN such as Cloudflare, export the request logs or query the analytics for bot traffic. The field names differ, but the filter is the same.
Verify the bot before you count it
A spoofed request is worse than no request, because it makes a blocked site look crawled.
- Reverse DNS. Resolve the IP to a hostname, then resolve the hostname back to the IP. Google publishes the
googlebot.comandgoogle.comdomains; Microsoft publishessearch.msn.com. A forward-confirmed reverse DNS lookup is the standard check. - Published IP ranges. OpenAI publishes its crawler IP ranges, and Google publishes its own in a JSON file. If the IP is outside the range, the user agent is a lie.
- Cloudflare verified bots. If the site is behind Cloudflare, its verified bot check does the work for you.
- Behavior. A real crawler requests HTML and assets at a steady rate. A spoofed one often hits the same URL repeatedly or probes admin paths.
An unverified request still matters for security, but it does not tell you whether OpenAI can read the page. Expect three outcomes, not two: verified, spoofed, and unverifiable when the operator publishes no address range to check against. The last one is not a failure, just missing evidence.
Run the analysis on a pasted log
You do not need to install anything to get the first numbers. Take an export of the log, cut it to the window you care about, and paste it into the analyzer.
- Export the access log for the period, or copy the last few thousand lines from the server.
- Keep the full lines. Verification needs the connecting IP, the user agent and the status, so trimming columns breaks the report.
- Paste the lines into the tool. It accepts combined, JSON, W3C extended and generic line formats, up to 750,000 characters per run.
- Read the verdict first, then the per-crawler table. The verdict names the problem; the table names the crawler and the page.
- Repeat monthly with the same window so the comparison means something.
A single day is too short. Crawlers arrive on their own schedule, so compare month to month, or week to week on a site that publishes often.
What to measure
| Metric | Question it answers |
|---|---|
| Requests per crawler | Is the bot visiting at all? |
| Verified vs spoofed | Is the traffic real? |
| Status codes | Is the bot getting 200, 403, 429 or 5xx? |
| Top paths | Which pages does the bot care about? |
| Bytes served | How much bandwidth does the crawler use? |
| Last seen | Is a crawler that used to visit now gone? |
The status codes are the most actionable. A crawler that receives 403 on every request is blocked by a firewall, even if robots.txt says allow. A crawler that receives 429 is being rate-limited. A crawler that receives 200 but only on the homepage may be following no internal links.
Read the result like a diagnosis
| Pattern | Likely cause | Fix |
|---|---|---|
| No AI crawler requests | robots.txt, firewall or no discovery | Check the robots.txt group and the WAF |
| Only spoofed requests | Real crawlers blocked before the origin | Check the CDN and bot management rules |
| 403 on article pages | A bot rule or login wall | Allow the crawler token on public content |
| 429 across the site | Rate limiting too aggressive for the crawl rate | Raise the limit or slow the crawler |
| Requests to the homepage only | Weak internal links or a JS-rendered nav | Add server-rendered links |
| Requests to old URLs | Stale sitemap or redirects | Update the sitemap |
Do not judge AI readiness from a single day. Crawlers visit on their own schedule, and a training crawler may fetch a page once a month.
Logs are necessary, not sufficient
A clean log tells you the crawler can reach the page. It does not tell you whether the page was used in an answer. Pair the log with a citation check: sample the questions your buyers ask and record which domains the AI answer cites.
Our AI bot log analyzer takes a pasted log, matches every known AI and search crawler token in it, checks each request against the operator's published address ranges, and reports hits, verified, spoofed and unverifiable counts, blocked and error responses, the top paths each crawler reads, and the last time it appeared. Start there, then read log file analysis for the wider crawl picture, the AI crawler list for the tokens, and how to block AI crawlers if the goal is the opposite.



