Configure robots.txt for AI crawlers

Before you start

• You can edit and deploy /robots.txt for every host you care about.
• You know whether your CDN or WAF also blocks bots. Google notes that crawling must be allowed “by any CDN or hosting infrastructure”, not just robots.txt.

1. Decide per purpose

Purpose	Token	If disallowed
Google Search, including AI Overviews & AI Mode	Googlebot	Page is not crawled for Search at all.
Gemini training and grounding; training of the models behind Search’s AI features	Google-Extended	No effect on Search inclusion or AI Overviews display.
ChatGPT search results	OAI-SearchBot	Not shown in ChatGPT search answers (may still appear as navigational links).
OpenAI model training	GPTBot	Content indicated as not for training.
User-initiated fetches in ChatGPT	ChatGPT-User	robots.txt “may not apply”; not used for search inclusion.
Claude search results	Claude-SearchBot	Content not indexed for Claude’s search; “may reduce your site’s visibility”.
Claude user requests	Claude-User	Claude cannot fetch your pages when users ask about them.
Anthropic model training	ClaudeBot	Future content excluded from training datasets.
Perplexity search results	PerplexityBot	Not surfaced in Perplexity search results.
User-initiated fetches in Perplexity	Perplexity-User	“Generally ignores robots.txt rules”, since a user asked.

OpenAI states that the OAI-SearchBot and GPTBot settings are independent. A site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Changes take about 24 hours to be reflected in search.

Officially documented · OpenAI

Anthropic separates ClaudeBot (training), Claude-User (user-requested fetches) and Claude-SearchBot (search indexing). Blocking ClaudeBot signals that future content “should be excluded from our AI model training datasets”. Blocking the other two may reduce visibility in Claude. Anthropic also honours the non-standard Crawl-delay directive, which Google does not support.

Officially documented · Anthropic

PerplexityBot surfaces and links websites in Perplexity search results and “is not used to crawl content for AI foundation models”. Perplexity-User fetches pages when users ask questions. Because a user requested the fetch, it generally ignores robots.txt rules. Changes may take up to 24 hours.

Officially documented · Perplexity

For Google, AI features are part of Search, so robots.txt rules for Googlebot are the crawl control. Google-Extended applies to some of Google’s other AI systems, not to Search.

Officially documented · Google Search Central

An August 2026 audit of robots.txt on 130 company sites looked at the companies that had made an explicit choice. They almost never blocked search crawlers and blocked training crawlers only. The study’s authors call blocking both together the most common and most damaging mistake.

Observed in practice · observed 2026-08 · DATA17 AI Research Lab · 解数智能 (unpublished report, not publicly available; readers can't check it)

2. Write the rules

Example: appear in Google, ChatGPT, Claude and Perplexity search; opt out of model training.

user-agent: Googlebot
allow: /

user-agent: OAI-SearchBot
allow: /

user-agent: GPTBot
disallow: /

user-agent: Google-Extended
disallow: /

user-agent: Claude-SearchBot
allow: /

user-agent: ClaudeBot
disallow: /

user-agent: PerplexityBot
allow: /

user-agent: *
allow: /

A crawler follows only the most specific group matching its name, so rules under * do not add to a named group.

3. Verify

1. Fetch the live file: curl -sA "OAI-SearchBot" https://example.com/robots.txt.
2. In Google Search Console, open the robots.txt report to confirm Google fetched the new version.
3. Check server or CDN logs for requests from each bot, and confirm the source IPs against the vendor’s published ranges (OpenAI publishes them as JSON for each bot).

Rollback

Restore the previous file. Google usually refreshes its cached copy within 24 hours.

Common mistakes

• Blocking OAI-SearchBot while meaning to block training.
• Assuming Google-Extended removes content from AI Overviews. It does not; use the Search generative AI control or snippet controls.
• Allowing bots in robots.txt while a WAF rule still returns 403.