AI crawlersยท
AI crawlers: every AI bot user agent and what it does
A list of AI crawlers from OpenAI, Anthropic, Google, Perplexity, Meta and others, with each bot's job, robots.txt behavior, IP list and rules to copy.
AI crawlers, robots.txt and llms.txt
Which bots read your site, how to allow or block them, and what llms.txt does.
Latest guides26 guides, newest first.
- AI crawlers
"Blocked by robots.txt" in Search Console: how to fix it
Blocked by robots.txt means a Disallow rule stops Googlebot crawling a URL. Find the rule, decide whether to unblock or noindex, then validate the fix.
- AI crawlers
Bytespider: what ByteDance's crawler does and how to stop it
Bytespider is ByteDance's web crawler. See its user agent, what ByteDance says it is for, the reports that it ignores robots.txt and how to block it.
- AI crawlers
CCBot and Common Crawl: should you block it?
CCBot is Common Crawl's crawler, and its archive feeds many AI training sets. See its user agent, how to block it, and what a block does and does not change.
- AI crawlers
How to track ChatGPT and AI traffic in GA4
Track GA4 AI traffic from ChatGPT, Perplexity, Gemini and Claude with the native AI Assistant channel, a source regex and a channel group above Referral.
- AI crawlers
Google-Extended: what it blocks and what it doesn't
Google-Extended controls Gemini training and grounding, not Google Search or AI Overviews. What the robots.txt token blocks and which control to use instead.
- AI crawlers
GPTBot: what OpenAI's crawlers do and how to allow or block them
GPTBot trains OpenAI's models, OAI-SearchBot feeds ChatGPT search and ChatGPT-User fetches for users. User agents, IP checks and robots.txt rules for each.
- AI crawlers
How to block AI crawlers with robots.txt, and when you shouldn't
How to block AI crawlers in robots.txt, with copy-paste rules to stop AI training, block one vendor or block every AI bot, and what each block costs.
- AI crawlers
llms.txt generator: how to create and publish llms.txt
How to create llms.txt with an llms.txt generator or by hand. The format, which pages to list, llms-full.txt, WordPress, Shopify and Next.js setup, and tests.
- AI crawlers
JavaScript SEO: why AI crawlers miss client-rendered content
JavaScript SEO for Google and AI crawlers. Why GPTBot, ClaudeBot and PerplexityBot miss client-rendered content, how to test a page and what to fix.
- AI crawlers
llms.txt examples from real websites, with a template
Real llms.txt examples from Svelte, Vercel, Cloudflare, Stripe, Anthropic and Perplexity, what each does well or badly, and a minimal template to copy.
- AI crawlers
Log file analysis for SEO: see which AI bots visit your site
Log file analysis for AI bots. Find your access log, count GPTBot, ClaudeBot and Googlebot hits with grep and awk, and catch fake crawlers by their IP.
- AI crawlers
Markdown vs HTML: what AI crawlers read best
Markdown vs HTML for AI crawlers. What Google says, which agents send Accept text/markdown, and how to serve Markdown without duplicate or cloaking issues.
- AI crawlers
Noindex in robots.txt: why it doesn't work and what to use
Noindex in robots.txt does nothing in Google. See why, how noindex vs disallow differ, and how to deindex a page with meta robots or X-Robots-Tag.
- AI crawlers
nosnippet, max-snippet and data-nosnippet: control what Google quotes
What nosnippet, max-snippet, data-nosnippet, max-image-preview and max-video-preview change in Google results, Discover and AI Overviews, with syntax.
- AI crawlers
PerplexityBot and Perplexity-User: what they do on your site
PerplexityBot indexes pages for Perplexity's answers and Perplexity-User fetches them for users. User agents, IP lists, robots.txt rules and blocking costs.
- AI crawlers
robots.txt Disallow: syntax, examples and common mistakes
What a robots.txt Disallow rule blocks, why one slash turns allow all into disallow all, and how to handle trailing slashes, query strings and Allow rules.
- AI crawlers
How to add your sitemap to robots.txt
How to write a robots.txt sitemap line that works, with exact syntax, sitemap index files, sitemaps on another host, CMS defaults and common mistakes.
- AI crawlers
robots.txt tester: how to test your robots.txt file
Google retired its robots.txt tester in 2023. How to test robots.txt now with Search Console, URL Inspection, Google's own parser, curl and AI crawlers.
- AI crawlers
How to block AI scrapers when robots.txt is not enough
How to block AI scrapers that ignore robots.txt with IP checks, Cloudflare AI bot policies, server rules and rate limits, and what each one costs you.
- AI crawlers
View rendered source vs raw HTML: what crawlers see on your page
How to view rendered source and raw HTML with view-source, curl, DevTools and Search Console, then compare them and see which crawlers run your JavaScript.
- AI crawlers
What is Googlebot and how does it crawl your site?
What is Googlebot? Its smartphone and desktop user agents, how it finds, crawls and renders pages, the 2 MB HTML limit, crawl rate and how to verify it.
- AI crawlers
What is robots.txt? A plain guide with examples
What is robots.txt? How the file works, its syntax, how crawlers settle Allow vs Disallow, what it can't do, and five annotated example files.
- AI crawlers
ClaudeBot: what Anthropic's crawler does on your site
ClaudeBot is Anthropic's training crawler. See its user agent, how it differs from Claude-SearchBot and Claude-User, and how to verify, slow or block it.
- AI crawlers
What is llms.txt? A complete guide with examples
What llms.txt is, the exact format with an example, llms-full.txt, which AI tools read it, what Google says about it, and how to publish one.
- AI crawlers
robots.txt for WordPress: the right setup
The robots.txt WordPress serves by default, how to edit it in Yoast, Rank Math or AIOSEO, and copy-paste rules for WooCommerce and AI crawlers.
Check how your site reads to search and AI engines
Enter a URL to see how ready the site is for search engines, answer engines and AI search, with the fixes in order.
Check your website