Exact robots.txt lines for each named AI crawler

Each block below is a complete, self-contained robots.txt group. Stack as many as you need in the same file. A crawler that matches a named group ignores your wildcard (User-agent: *) group entirely — the most specific matching group wins, groups are never combined.

Block a crawler entirely

User-agent: GPTBot
Disallow: /

Swap in any token from the table below. To block several with the same rule, list them as consecutive User-agent lines before the shared Disallow:

User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Explicitly allow a crawler even when your wildcard group blocks everything

User-agent: *
Disallow: /

User-agent: ClaudeBot
Allow: /

Known AI crawler tokens

This list is a snapshot summarized from public reporting and each operator's own documentation — operators add crawlers and change stated purposes over time. Check the AI Crawler Access Check tool against your current robots.txt, and confirm anything load-bearing directly with the operator.

What llms.txt is (and isn't)

llms.txt is an emerging, optional convention for a plain-text file at /llms.txt: an H1 title, an optional one-line blockquote summary, and Markdown-linked sections pointing an AI assistant at your most important pages. It is not a standard any crawler is documented to require, and it does not control crawling — robots.txt still does that. A minimal one looks like this:

# Your Site

> One sentence describing what this site is.

## Docs
- [Getting started](https://example.com/docs/start)
- [API reference](https://example.com/docs/api)

The AI Crawler Access Check tool checks the basic shape above and flags what's missing.

Common questions

How do I block GPTBot (OpenAI) from crawling my site?

Add a dedicated group to robots.txt: "User-agent: GPTBot" followed by "Disallow: /". A wildcard "User-agent: *" block does not cover GPTBot if a more specific GPTBot group also exists — the most specific matching group wins, not the sum of both.

How do I block every AI crawler at once?

There is no single standard token meaning "all AI crawlers" — each operator uses its own token. The reliable way is one Disallow: / group per named token you want blocked, or a broad "User-agent: *" / "Disallow: /" block if you intend to block all automated crawling, AI or not, which will also stop most search-engine indexing.

Does Google-Extended block Google Search from indexing my site?

No. Google-Extended controls only whether Google's generative-AI products (Gemini, Vertex AI) may use the content for training and grounding. Blocking it does not affect Googlebot or normal Search indexing, which is a separate user-agent.

What's the difference between GPTBot and ChatGPT-User?

GPTBot crawls to train OpenAI's models. ChatGPT-User fetches a specific page because a live ChatGPT user's session asked it to (browsing, plugins, actions). Blocking one does not block the other, and many sites intentionally allow ChatGPT-User while blocking GPTBot.

Do I need an llms.txt file?

No — it's an optional, emerging convention (see llmstxt.org), not a standard any crawler is documented to require. It's a plain-text file at /llms.txt giving an AI assistant a short map of your key pages: an H1 title, an optional one-line summary, and Markdown link sections.