AI discoverability
AI discoverabilityis making your site readable and quotable by AI systems — ChatGPT, Claude, Perplexity, Copilot and Google’s AI Overviews — so they cite you when someone asks a question your site answers. Traditional SEO gets you ranked; this gets you cited, which is a different job with a different checklist. A page that ranks third can be the one an engine quotes, and a page that ranks first can be skipped entirely if nothing on it is extractable.
Which crawlers can reach you?
Every AI engine crawls with its own user-agent, and each one you block is an engine that can never cite you. This is the cheapest thing to get wrong and the cheapest to fix.
| User-agent | Used by | Blocking it means |
|---|---|---|
GPTBot, ChatGPT-User | ChatGPT | No ChatGPT citations |
ClaudeBot | Claude | No Claude citations |
PerplexityBot | Perplexity | No Perplexity citations |
Google-Extended | Gemini, AI Overviews | Excluded from Google’s AI answers |
CCBot | Common Crawl (training corpora) | Not used for training — citation unaffected |
CCBot is the one genuine choice here: blocking it keeps your content out of training corpora while the search-and-cite bots above still work. Blocking the others trades away citations.
# Allow the crawlers that read your site to answer questions and cite you.
# Each engine uses its own agent, so one blocked line = one engine that can
# never cite you.
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# Optional: block training-only crawling while still allowing the
# search-and-cite bots above.
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlIs every public page actually reachable?
Worth checking before anything else, because it is the failure that hides all the others. An app that gates unknown paths behind a login will happily redirect a crawler to a sign-in page and return HTTP 200 — so monitoring looks green while the page is invisible. We shipped exactly this bug: our own /llms.txt and our version-pinned install URLs redirected to /login for weeks before anyone noticed, because nothing fails loudly when it happens.
Fetch each public URL with curl, unauthenticated, and confirm a real 200 with real content. Do the same for anything machine-readable you publish — a file nobody can fetch is worse than no file, because you will believe it is working.
Publish an llms.txt
llms.txt is a plain-text map of your site at /llms.txt: what the product is, who it is for, and where the useful pages are. It is not a ranking signal and Google does not require one — its value is that non-Google engines and coding agents read it, and it is the one place you get to state your own category membership in prose you control.
Three things matter more than length:
- State the category.“An invoicing tool for freelancers” is retrievable; “the last tool you’ll ever need” is not. An engine has to be able to place you in a set to include you in a comparison.
- Say what you are not for. Counter-intuitive, and the most-skipped line. Boundaries stop you being cited into comparisons you lose, and they make the rest of the description more credible.
- Generate the link list. If it is hand-maintained it will drift — ours did, silently, for six days after two new pages shipped. Build it from whatever array already enumerates your pages.
# Acme Invoicing
> Invoicing for freelancers who bill hourly.
Acme is an invoicing tool for independent contractors. It turns tracked
time into an invoice, chases payment, and files the VAT return. Free for
one client; usage-based above that.
It is not an accounting ledger and does not do payroll.
Docs last reviewed 2026-07-26.
## Docs
- [Quickstart](https://example.com/docs/quickstart): create your first invoice in under five minutes.
- [Time tracking](https://example.com/docs/time): start a timer, or import hours from a CSV.
- [VAT and tax](https://example.com/docs/vat): how Acme calculates VAT per jurisdiction.
## API
- [REST API](https://example.com/docs/api): create invoices and read payment status.
- [OpenAPI spec](https://example.com/openapi.json): machine-readable description of every endpoint.Write passages an engine can lift
AI systems quote passages, not pages. The unit of work is a paragraph that still makes sense with everything around it deleted.
- Define the subject, not yourself.Open with “Revenue per visitor is a channel’s revenue divided by its visitors”, not “We show you revenue per visitor”. The first is quotable in an answer about the concept; the second is only quotable in an answer about you, which is a much rarer question.
- Answer first, context after.A reader skims for the answer and an engine extracts the first one it finds. Roughly 40–60 words is the sweet spot for a lifted passage.
- Use tables for anything comparative. Options, parameters, plans, X-vs-Y. A prose bullet list carrying three facts per item is far harder to extract than three columns.
- Phrase headings the way people ask.“How do I exclude my own traffic?” beats “Exclusions”. Give every heading an
idso a specific passage can be linked. - Cite your sources. The strongest single lever in the published research on this — and if you are paraphrasing a law or a spec, name and link it rather than restating it anonymously.
- Show a date.Undated content loses to dated content, because recency is weighted. A visible “last reviewed” line plus
lastModifiedin your sitemap is enough.
Add structured data honestly
JSON-LD tells an engine what a page is. Google states plainly that it is not required for AI Overviews, so treat it as entity recognition for other engines plus ordinary SEO hygiene — worth doing, not worth agonising over.
Add Organization once, then whatever each page genuinely is: SoftwareApplication for a product page, TechArticle for docs, HowTo for a real ordered procedure. The discipline that matters is not adding the rest. FAQPage with invented questions, Review without reviews, an AggregateRating you made up — that is fabricated markup, it is a policy violation, and it is easy to detect. If the content is not on the page, it does not go in the markup.
Verify it
All four of these should be checked from outside your app, with no session and no JavaScript — the way a crawler sees you.
# Is your llms.txt actually reachable? (a login redirect is the classic trap)
curl -sI https://example.com/llms.txt | head -1
# Does a crawler see your prose, or an empty JS shell?
curl -s https://example.com/docs/quickstart | grep -c "the first sentence of your page"
# Any AI bot blocked?
curl -s https://example.com/robots.txt | grep -iE "gptbot|claudebot|perplexity|google-extended"
# Do you emit structured data?
curl -s https://example.com/ | grep -c 'application/ld+json'What actively hurts
- Keyword stuffing. In traditional SEO it is merely ineffective; for AI visibility it measurably reduces citation rates.
- Separate copy written for AI. Google names this as scaled content abuse. One set of pages, written for people.
- Chopping pages into fragments in the hope of being more extractable. Normal headings and paragraphs are what you want.
- Gating your best material. An engine cannot cite what it cannot fetch, so anything behind a login or an email form is invisible.
Hand this to your agent
If you would rather not do it by hand, this prompt covers everything above in the order that surfaces the expensive problems first. It is deliberately explicit about what not to do, because that is where an enthusiastic agent will otherwise invent schema.
Audit this site for AI-search citability and fix what you find. Work through it in this order:
1. robots.txt — make sure GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and
Google-Extended are not disallowed. Report any that are.
2. Fetch every public URL and confirm each returns 200 for an anonymous
request. Flag anything that redirects to a login page — that is the most
common cause of an invisible site.
3. Confirm the page prose is present in the initial HTML, not injected by
JavaScript. Fetch with curl (no JS) and check the body text is there.
4. Add or fix /llms.txt following llmstxt.org: an H1 with the product name, a
one-line blockquote summary, a short paragraph stating the category, the
audience and what the product is NOT for, then link sections. Generate the
link list from whatever already enumerates your pages so it cannot drift.
5. For each key page: does it open with a self-contained definition someone
could quote without the surrounding page? If it opens with "We ..." or
"Our product ...", rewrite the first sentence to define the subject instead.
6. Convert option/parameter/comparison lists into real HTML tables.
7. Give every H2 an id so passages can be deep-linked.
8. Add a visible "last reviewed" date and a lastModified in the sitemap.
9. Add JSON-LD: Organization plus whatever the page genuinely is (SoftwareApplication,
TechArticle, HowTo). Do NOT add FAQPage unless real question-shaped headings
already exist, and do NOT invent reviews or ratings.
10. Cite external authorities where you make a factual or legal claim.
Rules: do not write separate copy aimed at AI, do not keyword-stuff (it measurably
reduces AI visibility), and do not fabricate structured data. Report what you changed
and what you deliberately left alone.Measuring whether it worked
There is no AI-specific report in Search Console, so referral traffic is the practical signal: visits arriving from chatgpt.com, perplexity.ai and friends. In Statlark those show up as regular referrer sources on your Overview — and because sources carry revenue, you can see whether AI-referred visitors actually buy, not just arrive. Pair that with a monthly manual check: ask each engine the ten questions your site should answer, and record who gets cited.
Last reviewed