AI crawler checker: access, rendering and real visits in three runs
Whether GPTBot, ClaudeBot and PerplexityBot may fetch a page, what they see without JavaScript, and whether they really come.
By ClaudeGPT Si Newsroom, Editorial team··3 min read
An AI crawler checker should answer three questions about your site: may AI bots fetch it, can they read it once they do, and are they actually coming. seogeoaeo.ai splits those into three single-purpose runs, starting with the AI Crawler View, which shows a page the way GPTBot, ClaudeBot and PerplexityBot see it.
Why one check is not enough
Most "is my site visible to AI" problems come from one of three places, and each needs a different test.
- Access. A line in robots.txt tells a bot to stay away. The page never gets fetched.
- Rendering. The bot fetches the page but gets an almost empty HTML shell, because the text, prices or headings are added by JavaScript in the browser.
- Reality. The rules look fine, yet the bot never visits, a firewall turns it away, or something pretending to be GPTBot is hitting your server.
A single green tick for "AI crawlers allowed" covers only the first one.
The three runs
| Question | seogeoaeo.ai tool | What you get | Price |
|---|---|---|---|
| Can the page be read without JavaScript? | AI Crawler View | Raw HTML compared with the rendered page: content, headings, prices, structured data, links and metadata that only appear after scripts run | 8 credits |
| Does robots.txt allow the bot? | Robots.txt & AI Crawler Checker | Access for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, the rule that matches, the line to change | 10 credits |
| Which bots really visit? | AI Bot Log Analyzer | From a pasted access log: which AI crawlers came, what they read, what failed, which visits used a bot's name from an unpublished IP | 8 credits |
Each run shows its credit price before it starts, and an AI agent can call the same tools over REST or MCP.
Start with rendering
Rendering is the problem people check least. The seogeoaeo.ai tool page notes that most AI crawlers, GPTBot, ClaudeBot and PerplexityBot included, read the HTML your server returns and do not run JavaScript. A single-page app can look perfect in Chrome and still hand those bots a blank div.
The AI Crawler View loads the raw HTML, renders the same URL in a headless browser, and lists what exists only in the second version. If your product description or pricing table shows up in that list, an AI answer engine is unlikely to quote it. The fix is server-side rendering or static generation, so the content ships in the first response.
Then check access
Open the robots.txt check with no path to see every rule that blocks a search or AI crawler. Then test the pages you care about with a specific token, such as GPTBot or ClaudeBot. A bot follows only the most specific group that names it, so a User-agent: GPTBot group overrides whatever you wrote under User-agent: *. That catches the common case where someone allowed everything and then added a narrower group that blocks one bot.
Finish with the logs
Rules are requests. The robots.txt check cannot prove that a crawler obeys them, and it cannot see a firewall or bot-protection setting that returns errors to AI bots. Paste a slice of your access log into the log analyzer to see what the bots received. OpenAI, Anthropic and Perplexity publish IP lists for their crawlers, which is how a visit claiming to be GPTBot can be checked.
When something else fits better
If you need a full crawl of thousands of URLs or ongoing rank tracking, a desktop crawler or an SEO suite does that job; seogeoaeo.ai checks one URL per run. For the question "can AI bots get to and read this page," the three runs above give a direct answer at a known cost.