Do AI crawlers run JavaScript?

Most AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, read the HTML the server returns without running JavaScript, so content that appears only after scripts run can be invisible to them.

How do I fix content that needs JavaScript?

Render it on the server, with server-side rendering or static generation, so it is in the HTML the server returns.

Does it check robots.txt access?

No. Use the Robots.txt & AI Crawler Checker for access rules, and the AI Bot Log Analyzer to see what bots actually fetched.

How much does the AI Crawler View cost?

8 credits per run, shown before the run starts. An AI agent can run the same tool over the REST API or MCP with a workspace API key.

Pricing

GEO · Rendering and mobile · 8 credits

AI Crawler View: Raw vs Rendered HTML Check

Find the content AI crawlers miss because it loads with JavaScript.

AI Crawler View 1.2.0 · price g32-v1 · retrieved at run time · Run it from an AI agent

What the AI Crawler View checks

See a page the way GPTBot, ClaudeBot and PerplexityBot see it. The raw HTML is compared with the page rendered in a headless browser, to find content, headings, prices, structured data, links and metadata that only appear after JavaScript runs.

The raw HTML compared with the rendered page
Content and headings that need JavaScript
Prices, structured data, links and metadata added after load

What it does not do

Checking whether robots.txt allows AI crawlers (use the Robots.txt & AI Crawler Checker)
Proving what a specific bot fetched (use the AI Bot Log Analyzer)
Visual layout or screenshot comparison

Questions about the AI Crawler View

Do AI crawlers run JavaScript?: Most AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, read the HTML the server returns without running JavaScript, so content that appears only after scripts run can be invisible to them.
How do I fix content that needs JavaScript?: Render it on the server, with server-side rendering or static generation, so it is in the HTML the server returns.
Does it check robots.txt access?: No. Use the Robots.txt & AI Crawler Checker for access rules, and the AI Bot Log Analyzer to see what bots actually fetched.
How much does the AI Crawler View cost?: 8 credits per run, shown before the run starts. An AI agent can run the same tool over the REST API or MCP with a workspace API key.