AI search engine optimization: a step-by-step playbook

AI search engine optimization is the work of getting your pages fetched, retrieved and cited by ChatGPT search, Perplexity, Claude, Microsoft Copilot and Google's AI Overviews and AI Mode. It comes down to seven jobs, done in order. Let each AI search crawler in, serve your content in raw HTML, get indexed by Google and Bing, make your brand one clear entity, lead every section with its answer, earn mentions on the sites AI answers already cite, and measure what comes back.

Each step depends on the one before it. A perfect answer paragraph does nothing if OAI-SearchBot gets a 403 from your firewall. Google's guide to generative AI search calls this work "still SEO" and says you need no special AI files or markup. That holds for Google, but ChatGPT, Perplexity and Claude send their own crawlers, and sites block those without noticing.

1. Allow AI search crawlers in robots.txt and at your CDN

Each AI search engine fetches pages with its own named bot, and these five decide whether a page can appear in AI answers.

The training crawlers, GPTBot and ClaudeBot, are a separate choice, and blocking them does not remove you from ChatGPT or Claude search. Our ClaudeBot guide explains that split for Anthropic's bots.

Google-Extended is the token people get wrong. Google's crawler documentation says it covers Gemini training and grounding, not Google Search. AI Overviews and AI Mode follow Googlebot rules, so blocking Google-Extended won't remove you from them.

This robots.txt lets the search bots in and keeps the training crawlers out.

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /checkout/

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /checkout/

The /checkout/ rule appears twice on purpose. A bot that finds a group naming it follows only that group and skips User-agent: *, so repeat the rules you still want.

Robots.txt is a request, and your CDN or firewall can still refuse a bot it allows. Cloudflare's AI bot policies sort bots into Search, Agent and Training, where Agent means bots acting for a person in real time, such as chat fetch bots. Since September 15, 2026, new Cloudflare domains block Training and Agent bots by default on pages that show ads. Perplexity says sites behind a WAF may need to allow its bots explicitly. For user-triggered fetchers the firewall is the real gate, since OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it.

Our robots.txt checker shows which rule matches each AI crawler for any path. It doesn't impersonate those bots, so it can't see a CDN rule aimed at them. For that, paste a CDN or server log into the AI Bot Log Analyzer, which lists AI crawler requests refused with 401, 403 or 429.

2. Serve your content in raw HTML for AI crawlers

Put every answer, price and heading in the HTML your server returns, because most AI crawlers don't run JavaScript. When Vercel and MERJ analyzed AI crawler traffic in December 2024, none of the major AI crawlers rendered JavaScript, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot included. Gemini was an exception, since it runs on Googlebot's infrastructure.

A React storefront that loads product details in the browser shows Googlebot a full page and shows OAI-SearchBot an empty <div id="root">. Server-side rendering or static generation fixes it.

To test, open view-source: in Chrome and search for a sentence from mid-article. If it's missing, AI crawlers miss it too. AI Crawler View runs the full comparison and lists the headings, prices, structured data, links and metadata that appear only after JavaScript runs.

3. Get indexed by Google and Bing

AI search engines answer from search indexes, so a page missing from the index is missing from the answers. Google's AI features documentation says a page needs only to be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. OpenAI says ChatGPT search rewrites a prompt into targeted queries and sends them to third-party search providers.

1. Verify the site in Google Search Console and Bing Webmaster Tools, and submit your XML sitemap to both.
2. Search your HTML and headers for noindex, nosnippet and max-snippet, which limit what Google's AI features can show.
3. Check Settings, then Search generative AI, in Search Console. Exclude keeps the whole site out of AI Overviews, AI Mode and Discover's generative AI features.

Our indexing checker reads the status code, robots.txt, meta robots, X-Robots-Tag, the canonical and snippet controls for Google and Bing, then names the signal that blocks the page. It can't confirm a URL is in Google's index, which is URL Inspection's job. For the Google side in depth, read how to show up in Google AI Overviews.

4. Make your brand one unambiguous entity

AI answers name companies they can identify, so give every engine the same name and the same profiles. Google's Organization markup documentation says the markup helps Google disambiguate your organization in search results, and it belongs on the home page or an about page.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Northwind Payroll",
  "url": "https://northwindpayroll.example",
  "logo": "https://northwindpayroll.example/logo.png",
  "sameAs": [
    "https://www.linkedin.com/company/northwind-payroll",
    "https://www.crunchbase.com/organization/northwind-payroll"
  ]
}

Google's AI guide says structured data isn't required for generative AI search, so treat this block as a statement of facts, not a ranking switch. The harder problem is name drift, where "Northwind", "Northwind Payroll Inc." and "NW Payroll" show up across title tags, og:site_name and directory listings. Pick one name and use it everywhere.

Brand Entity Checker looks for one name across your Organization and WebSite markup, og:site_name, the web manifest and llms.txt. It also checks that the logo loads, that sameAs profiles exist and whether Wikidata lists your site.

5. Put the answer first in each section

Open each section with a sentence that answers its heading on its own, then add the detail. A sentence that makes sense out of context can be quoted, and one that opens with "It depends" or "As mentioned above" can't.

Our guide to optimizing content for AI search covers passage-level writing. Citability Analyzer reviews each section of a page for a clear answer, context and supporting evidence. Treat its score as an editorial aid, not a citation forecast.

6. Earn mentions on the sites AI answers already cite

Find the pages AI answers cite for your category, then get your brand onto them. An engine can name you from a comparison article or a forum thread without ever linking your site.

Send comparison article authors current pricing and a demo account, and ask for a fix when they list an old price. On review sites and forums, claim your profile and answer threads under your own name. Our post on how to get cited by ChatGPT covers that outreach.

AI Citation Finder finds the targets. Enter a topic and it lists the 25 pages cited in the most Google AI Overviews and ChatGPT answers, and the sites cited most. That is your outreach list.

7. Measure AI search engine optimization results

Use several sources, because no single report covers every engine.

• Search Console's Generative AI performance report, open to all sites since August 31, 2026, shows impressions only, by page, country, device and date, with AI Overviews and AI Mode combined.
• Bing Webmaster Tools' AI Performance report, launched in public preview on February 10, 2026, counts citations in Microsoft Copilot, Bing's AI summaries and some partner integrations, with a sample of grounding queries.
• In analytics, segment sessions from chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com. Some apps send no referrer, so treat the count as a floor. Our AI referral tracking guide has filters for GA4 and other tools.

To see the answers themselves, sample them. AI Answer Visibility asks ChatGPT, Perplexity and Gemini one of your customers' questions with web search on, and reports whether each answer names or cites your brand and which brands it names instead. Answers vary between runs, so ask the same questions each month and watch the trend.

Check the first five steps on your own site

Our website check runs most of steps 1 to 5 on the pages you choose. It reads robots.txt rules for AI crawlers, indexability and snippet controls, compares raw and rendered HTML, reviews citability and checks your brand's entity signals. You get one prioritized fix list scored from 0 to 100, and a rerun shows the percent of issues you fixed.

The full check is priced in credits by the checks and pages you pick, and you see the total before it runs. A quick check of your home page costs no credits and takes about 20 seconds. It runs the crawling and indexing, on-page SEO and structured data checks on that page.