LLM SEO: how to get your site found and cited by AI search

LLM SEO is the work of making your pages reachable, readable and quotable for the search systems behind AI answers, so ChatGPT, Perplexity, Claude, Copilot and Google's AI features can find them and cite them. Most of it is technical SEO you may already do. Three things are different: each AI engine has its own crawler, several of those crawlers do not run JavaScript, and success shows up as mentions and citations rather than a rank.

This guide covers the technical side for developers. For what the terms mean and how they overlap, read SEO vs AEO vs GEO.

Quick summary

Allow each engine's search crawler in robots.txt, and decide separately whether to allow its training crawler. Make sure the facts on a page are in the HTML the server sends, because several AI crawlers do not execute scripts. Keep pages indexable and eligible for snippets in Google and Bing, then check your server logs for pages actually served to crawlers and track whether answers mention and cite you.

How AI search engines find your pages

An AI answer draws on what the model learned in training and on pages a search system fetches when someone asks. You cannot edit the first. LLM SEO works on the second, which depends on crawlers whose access you control.

The two paths reach your site through different crawlers:

Each engine documents its own route. Read on October 14, 2026:

OpenAI, Perplexity and Anthropic also run a separate agent that fetches a page while a user waits for an answer: ChatGPT-User, Perplexity-User and Claude-User. OpenAI says robots.txt rules may not apply to these user-triggered visits, and Perplexity says its user agent generally ignores them. Anthropic says all three of its bots follow robots.txt.

Step 1: Let the search crawlers in

Search crawlers and training crawlers use different robots.txt names, and the settings are independent. Blocking GPTBot keeps your content out of OpenAI's model training, but it does not remove you from ChatGPT search. Blocking OAI-SearchBot does. OpenAI says sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links.

The same split exists at Anthropic, where ClaudeBot collects training data and Claude-SearchBot indexes pages for search. At Google, Google-Extended is a robots.txt token with no crawler of its own. It controls whether Google may use your content to train Gemini models and to ground answers in Gemini Apps, and Google says it does not affect inclusion or ranking in Google Search.

This robots.txt allows the search crawlers and opts out of training. The training groups are a choice, not a requirement. Delete them if you are happy for your content to be used in training, and keep in mind that blocking Google-Extended also covers grounding in Gemini Apps.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

OpenAI and Perplexity both say a robots.txt change can take up to about 24 hours to reach their systems. robots.txt is also not the only gate. A firewall or CDN bot rule can block a crawler before it ever reads the file, so check those rules too. OpenAI, Perplexity and Anthropic publish their crawler IP addresses, linked from the pages above, which lets you allow them explicitly.

Step 2: Put the content in the HTML

In December 2024, Vercel analyzed AI crawler traffic on its network and found that none of the major AI crawlers rendered JavaScript. ChatGPT's and Claude's crawlers fetched JavaScript files but did not execute them. Gemini was the exception, because it uses Googlebot's rendering. Crawlers can change after a study like this, but HTML rendered on the server works for all of them either way.

So if a page builds its main text in the browser, an AI crawler may see an empty shell. Render the text, prices, specs and comparisons that buyers ask about on the server, or prerender the page.

You can check what a crawler without JavaScript receives with curl, which does not run scripts either. Replace the URL with one of your pages and the text with a sentence you see on that page in a browser. The -A flag sends a crawler-style user agent, in case your server treats bots differently.

url="https://example.com/pricing"
text="Billed per request"
result=$(curl -sS -A "OAI-SearchBot/1.4" -o page.html \
  -w '%{http_code} %{redirect_url}' "$url")
status=${result%% *}
case "$status" in
  200) grep -qF "$text" page.html && echo "Found in the HTML" || echo "Not in the HTML" ;;
  3??) echo "Redirects to ${result#* }; check that URL instead" ;;
  000) echo "No response from $url" ;;
  *) echo "HTTP $status: the server refused or failed the request" ;;
esac

Only a 200 answer tells you anything about rendering. A redirect, a 403 or no response means the crawler never got the page, so fix the URL, firewall or bot rule first. If the result is "Not in the HTML", open page.html to confirm it is your page and not a bot challenge or error page. Then repeat the check for each template, such as product, pricing, docs and blog pages, rather than only the home page.

Step 3: Stay eligible in Google and Bing

Google's AI features documentation says a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode, and that there are no additional technical requirements. The snippet controls apply there too. A nosnippet or max-snippet rule, or a data-nosnippet attribute, limits what Google can show from the page in AI features as well as in ordinary results.

For Copilot and Bing, the Bing Webmaster team says freshness signals affect how quickly updates reach search results and AI-generated answers. Keep a sitemap whose lastmod values are the real time each page last changed, in ISO 8601 with a time, and not the time the sitemap was generated. Bing also accepts IndexNow pings when a URL changes.

Step 4: Make the facts quotable

An engine can only cite what it can read on a page it fetched. Put the facts people ask about, such as what the product does, who it is for, prices and limits, in plain text on a crawlable URL, not only inside images, PDFs or interactive widgets.

You do not need a special format. Google's AI optimization guide says you don't need to write in a specific way for generative AI search, and that there is no requirement to break content into small chunks. Clear headings and a direct answer near the top help readers, and that is reason enough.

What about llms.txt?

llms.txt is a proposal for a Markdown file at your site root that points AI agents and tools to your most useful pages, and some documentation platforms generate one automatically. The proposal page describes the file format. Google's AI optimization guide says you don't need AI text files such as llms.txt to appear in Google Search, because Google Search does not use them. Publish one if agents that read your docs benefit from it, not as a ranking step.

Step 5: Confirm crawlers get your pages

Your access logs show whether the crawlers reach you, but a request is not the same as a page served. This loop reads an access log in the combined format that nginx and Apache use by default. For each crawler it counts every request, and separately the requests for pages other than robots.txt that got a 200 or 304 response:

for bot in OAI-SearchBot ChatGPT-User GPTBot PerplexityBot Perplexity-User \
  Claude-SearchBot Claude-User ClaudeBot bingbot Googlebot; do
  awk -v bot="$bot" 'index($0, bot) { all++; if ($7 != "/robots.txt" && ($9 == 200 || $9 == 304)) ok++ }
    END { printf "%6d requests %6d pages served  %s\n", all, ok, bot }' access.log
done

Then check the pages you care about. This lists the status codes OAI-SearchBot received for your pricing page:

grep "OAI-SearchBot" access.log | awk '$7 == "/pricing" { print $9 }' | sort | uniq -c

A crawler can only read a page that returned 200, or 304 when it already holds a copy. Requests with no pages served, only robots.txt requests, or 403 and 5xx responses on your key pages can point to a firewall rule, a CDN bot setting or a robots.txt group you did not intend. User-agent strings can be faked, so for decisions that matter, compare the request IPs with the addresses each operator publishes.

Step 6: Measure mentions and citations

Pages served with a 200 show that a crawler could fetch them. They do not show whether answers mention or cite you. Google's Search Console reports impressions of your links in AI Overviews and AI Mode, and Bing Webmaster Tools reports citations in Copilot in a preview report. For ChatGPT, Claude and Perplexity, send a fixed set of buyer questions on a schedule and record whether each answer mentions your brand and which pages it cites.

The measurement section of SEO vs AEO vs GEO lists each report, and the AI search visibility guide explains how to build the question set. Keep the questions, engines and country the same between rounds, so a change after a site update is easier to spot. AI answers also vary from run to run, so treat one round as a sample, not proof.

Next step

Open your robots.txt and check each search crawler from the table above. Then run the curl check on your three most important page templates. If a page comes back with a 200 and a key sentence is still missing from its HTML, fix rendering before you change any copy.