Robots.txt checker: see which rule blocks a crawler and what to change
Test any path as Googlebot, GPTBot or another bot, read the matching rule, and fix the right line.
By ClaudeGPT Si Newsroom, Editorial team··3 min read
A robots.txt checker reads the file at the root of your site and tells you, for a given crawler and path, whether a fetch is allowed and which rule decided it. The seogeoaeo.ai Robots.txt & AI Crawler Checker does that for search bots and AI crawlers in one run and points to the exact line to change.
Why a checker beats reading the file by eye
A short robots.txt looks easy to read. The trouble starts when the file has several user-agent groups, wildcards and overlapping Allow and Disallow lines. Google picks the most specific rule, measured by the length of the path, and when two rules conflict it uses the less restrictive one. A crawler also follows only the group that names it most specifically, so a User-agent: Googlebot group means Googlebot ignores everything under User-agent: *. Those rules are simple on paper and easy to get wrong in a real file.
What the seogeoaeo.ai checker reports
You give it a public URL. Two inputs are optional: a path to test, such as /blog/post, and a crawler token, such as Googlebot or GPTBot. The run returns:
| Output | What it answers |
|---|---|
| Blocking rules | Which lines stop a search or AI crawler from fetching pages |
| Matching rule | The rule that applies to your path for the crawler you named |
| Crawler access | Whether Googlebot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others are allowed |
| Fix | The exact line to edit |
A run costs 10 credits, and the price shows before it starts. The same tool runs from the browser, or from an AI agent over the REST API or MCP.
A fix loop that works
- Run the checker on your home page with no path. Read the list of blocking rules first; an unexpected
Disallow: /under the wrong group is the most expensive mistake a site can make. - Test the paths that earn you money: product pages, pricing, the blog. Test each one as Googlebot and as
*. - Edit the line the report names, publish, and run the check again. The checker reads the live file, so it sees your change as soon as it is deployed.
- Remember that Google generally caches robots.txt for up to 24 hours, so crawl behavior can lag behind the file.
Rules to keep in mind while you fix
- The file must be called robots.txt, saved as UTF-8 text, and sit in the top-level directory.
- It only covers the host, protocol and port it is served from.
shop.example.comneeds its own file. - Paths are case-sensitive.
Disallow: /Admindoes not block/admin. *matches any run of characters and$marks the end of a URL.- Google ignores anything past 500 KiB.
- Blocking a page does not remove it from search. Google can still index a blocked URL and show it without a snippet. Use the Indexing & Canonical Checker when the question is about indexing rather than crawling.
What the checker does not do
It does not fetch or render the path you test, it does not validate your sitemap, and it cannot prove that a bot obeys the file. Robots.txt is a request, not an access control. It also needs a live, public URL. If you want to try edits before they go live, Bing Webmaster Tools has a tester with an editor, and Google publishes an open source parser you can run locally. For checking the published file and naming the line to change, seogeoaeo.ai is the faster pick.
Bottom line
Run a robots.txt checker after every change to the file and after every site migration. It takes one run to learn whether your key pages are open to the crawlers you care about, and the report tells you which line to edit when they are not.