GEO best practices for documentation sites and developer products

This guide is for people who run a documentation site, an API reference or a developer product. Developers now ask ChatGPT, Claude, Perplexity and coding assistants before they open your docs. The goal here is that those answers quote the current, correct page. Most of the work is a one-off engineering task of a few days, followed by habits in the release process.

The short version

1. Let AI search crawlers and user-triggered fetchers reach the docs.
2. Put the full text of every page in the server HTML.
3. Offer a clean Markdown version of each page.
4. Give each docs version one canonical URL and make “latest” obvious.
5. Write code examples and reference pages that survive being quoted out of context.
6. Publish changes in a dated, linkable changelog and keep sitemaps honest.
7. Add llms.txt if your platform makes it cheap, and don’t expect it to move rankings.

1. Let the right crawlers in

Docs sites often sit behind the same bot protection as the app. That is the most common reason a good page never shows up in AI answers. Two kinds of bots matter: search crawlers that index pages ahead of time, and user-triggered fetchers that load a page when someone asks about it.

OpenAI: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

Officially documented · OpenAI

Anthropic separates ClaudeBot (training), Claude-User (user-requested fetches) and Claude-SearchBot (search indexing). Blocking Claude-User or Claude-SearchBot may reduce visibility in Claude.

Officially documented · Anthropic

PerplexityBot surfaces and links websites in Perplexity search results. Perplexity-User fetches pages when users ask questions and generally ignores robots.txt rules, because a user requested the fetch.

Officially documented · Perplexity

• Allow the search tokens: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot.
• Decide on training crawlers (GPTBot, ClaudeBot, Google-Extended) separately. Blocking them does not remove you from search.
• Check the CDN or WAF, not only robots.txt. Token-by-token details are in Configure robots.txt for AI crawlers; the full check is in GEO best practices for technical and SEO leads.

2. Put the text in the HTML

Many docs frameworks render navigation and content in the browser. A crawler that does not run JavaScript then sees an empty shell.

Google recommends server-side or pre-rendering because “not all bots can run JavaScript”.

Officially documented · Google Search Central

Observed in 2024-12, more than 9 months ago. Answer engines change often; re-test before relying on this.

In tests published in December 2024, Vercel and MERJ found that none of the major AI crawlers they examined rendered JavaScript, including OpenAI’s, Anthropic’s and Perplexity’s. Gemini, which uses Googlebot’s infrastructure, did.

Observed in practice · observed 2024-12 · Vercel & MERJ

Check with curl -s <url> and search the output for a sentence from the body and a line of a code sample. Tabs, accordions and “copy” widgets should keep their content in the HTML, hidden with CSS rather than fetched on click. More in JavaScript rendering and AI crawlers and JavaScript sites (Next.js, Vue and others): making sure AI crawlers can read your content.

If curl can’t see your code sample, neither can most AI crawlers.

3. Offer a Markdown version of each page

Markdown removes navigation, scripts and layout, so an assistant gets only the content. Several docs platforms now generate it automatically.

Mintlify automatically hosts llms.txt and llms-full.txt at the root of a docs project. “Page links in the llms.txt file include a .md extension so AI tools can fetch the Markdown version of each page directly.”

Officially documented · Mintlify

GitBook serves /llms.txt, which lists every published page with a Markdown version, and /llms-full.txt with the full documentation in one file. Appending .md to any published page URL returns that page as Markdown.

Officially documented · GitBook

If you build your own site, follow the same pattern: page-url.md returns the source Markdown with headings and fenced code blocks intact. Keep it identical to the HTML page. A Markdown copy that drifts from the HTML is a second, stale version of your docs. Add a “Copy as Markdown” or “Open in ChatGPT/Claude” button only if it uses the same file.

4. One canonical URL per version

Versioned docs create near-duplicate pages: /v2/auth, /v3/auth, /latest/auth. If an assistant quotes the wrong one, users copy deprecated code.

Google ranks canonical signals by strength. Redirects and rel="canonical" are strong signals, and sitemap inclusion is weak. Combining them increases the chance your preferred URL is chosen.

Officially documented · Google Search Central

• Pick one stable URL for the current version (for example /docs/auth) and point rel="canonical" there from /latest/ aliases.
• Keep old versions online if users need them, but say so in the page title and first line: “Version 2 (deprecated). Current: v3.”
• List only current-version pages in the sitemap and in llms.txt.
• Put the version number and “last updated” date in visible text, not only in a dropdown.

Background: Canonical URL signals.

5. Write pages that survive being quoted

AI answers lift a paragraph or a code block and drop the rest. This section is our editorial advice, based on how docs get quoted, not on any platform’s documentation.

• Make every code example complete: imports, package name, install command and version.
• State prerequisites and the version near the top of each page, not only on an intro page.
• Use one name per concept. If the SDK method is createClient, don’t call it “the init call” elsewhere.
• Give each error message its own heading or table row, with the exact error string. Developers paste errors into assistants.
• Write a one-sentence description under each H1 that says what the page is for.

The general writing guidance is in For content editors: writing articles AI answers can quote and Claim: content must be chunked and rewritten for AI search.

Docs get quoted one block at a time. Each code sample should run on its own.

6. Date your changes and keep sitemaps honest

When an API changes, the old answer lives on in assistants until the new page is crawled again. A dated changelog with one URL per release gives crawlers and users a single place to learn what changed.

Google ignores <priority> and <changefreq> values. It uses <lastmod> “if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate”.

Officially documented · Google Search Central

• Give each release note its own anchor or page, with the date and affected versions.
• Link from the changelog entry to the updated reference page, and back.
• Set <lastmod> only when the content changes, not on every build. See XML sitemaps and XML sitemaps vs IndexNow.
• When you deprecate something, edit the old page to say so. Don’t just delete it.

7. llms.txt: useful, but not a ranking lever

llms.txt is a proposed index file for language models. For docs it is a reasonable convenience, especially when coding tools or users load it on purpose. It is not a way to rank in AI search.

/llms.txt is a proposal (llmstxt.org, 2024) for a Markdown file at a site’s root. It gives a short summary of the site and links to key pages, optionally with Markdown versions. The aim is to help language models use the site at inference time.

Officially documented · llmstxt.org

Google Search does not use llms.txt files. Maintaining them for other services “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”

Officially documented · Google Search Central

A 2026 study by DATA17 checked llms.txt on 163 brand domains in one product category. Brands one AI assistant mentioned had deployed llms.txt less often (17%) than brands it never mentioned (27%). With only 18 brands mentioned, the authors read this as no measurable positive effect.

Observed in practice · observed 2026-08 · DATA17 AI Research Lab · 解数智能 (unpublished report, not publicly available; readers can't check it)

If your platform generates it, leave it on. If you build it by hand, generate it from the same source as the sitemap so it can’t go stale. Full evidence: Claim: llms.txt improves AI search visibility.

Ship llms.txt for the developers who ask for it. Don’t ship it expecting citations.

What not to do

• Don’t block all bots at the docs CDN. Search and user-fetch crawlers are separate from training crawlers (Configure robots.txt for AI crawlers).
• Don’t serve different content to AI bots. Keep the HTML, Markdown and llms.txt in sync from one source. Google treats creating separate content mainly to game AI answers as risky (Scaled content abuse).
• Don’t let /latest/ and /vN/ compete. Choose a canonical (section 4).
• Don’t put examples only in images or interactive sandboxes. Keep a text version on the page.
• Don’t treat llms.txt as the GEO project. It is one small item (Claim: llms.txt improves AI search visibility).

How you’ll know it’s working

• Logs: count verified visits from OAI-SearchBot, Claude-SearchBot, PerplexityBot and user-fetch agents to docs URLs, including .md URLs (Verify AI crawler traffic in server logs).
• Bing AI Performance: see which docs pages Copilot and Bing AI cite (Bing AI Performance report).

Microsoft: “AI Performance does not measure rankings, authority, performance, or importance. It simply shows which content was cited.”

Officially documented · Microsoft Bing

• Prompt panel: keep 20–30 real developer questions (“how do I paginate with the X SDK”) and run them monthly. Check whether answers cite your docs and use the current version (Measuring AI visibility by sampling prompts).
• Support tickets: tag tickets where the user followed an outdated AI answer. A falling count is the outcome that matters.

Evidence at a glance

Recommendation	Evidence
Allow search and user-fetch crawlers	Official (OpenAI, Anthropic, Perplexity)
Server-rendered HTML	Official (Google) and observed (Vercel/MERJ, 2024)
Markdown version of each page	Official platform features (Mintlify, GitBook); effect on citations untested
One canonical per version	Official (Google canonical signals)
Self-contained code examples and error pages	Hypothesis (our editorial advice)
Accurate lastmod and dated changelog	Official (Google sitemaps) for lastmod; changelog structure is our advice
llms.txt	Official: ignored by Google Search; observed: no measurable effect in one study