SEO · Technical page check · 10 credits
Robots.txt Checker & Tester for AI Crawlers
See every robots.txt rule that blocks search or AI crawlers, the rule that matches a given path, and the exact line to change.
Robots.txt & AI Crawler Checker 1.0.0 · price a38-v1 · retrieved at run time · Run it from an AI agent
Loading tool
What the Robots.txt & AI Crawler Checker checks
Audit a site's robots.txt, test whether a crawler may fetch a path, and see which search and AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more) are allowed.
- Every rule that blocks a search or AI crawler
- The rule that matches a given path for a given crawler
- Access for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more
- The exact line to change
What it does not do
- Rendering or fetching the tested path itself
- Sitemap validation or indexation claims
- Proof that a crawler obeys robots.txt
Questions about the Robots.txt & AI Crawler Checker
- How do I block or allow GPTBot in robots.txt?
- Add a group for its user agent: "User-agent: GPTBot" followed by "Disallow: /" to block it or "Allow: /" to allow it. A crawler follows only the most specific group that names it, so a GPTBot group overrides the "User-agent: *" rules. Run the checker after you publish to confirm which rule now matches.
- What does Google-Extended control?
- Google-Extended is a robots.txt token that controls whether Google may use your content for its Gemini models and grounding. It does not change how Googlebot crawls or ranks your pages in Google Search.
- Do AI crawlers obey robots.txt?
- The major operators say their crawlers follow it, but robots.txt is a request, not an access control, and this checker cannot prove what a crawler does. To see what bots actually fetch, paste your access log into the AI Bot Log Analyzer.
- How much does the Robots.txt & AI Crawler Checker cost?
- 10 credits per run, shown before the run starts. An AI agent can run the same tool over the REST API or MCP with a workspace API key.
