Which AI crawlers can actually read your site?

Paste your robots.txt and this checks it against 20 named AI-crawler user-agents — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more — so you can see in one place who's allowed in, who's blocked, and who falls through to your wildcard rule by accident.

Nothing is uploaded. The text you paste is parsed in this browser and never leaves your device.

1. Try fetching it automatically, or paste it

Domain (optional — many sites block cross-site requests to static files, so this won't always work)

robots.txt content llms.txt content (optional)

Prefer a CLI, or want this as a CI gate?

Same logic, zero dependencies, one file — and it can fetch a live domain directly (Node has no cross-origin restriction, unlike this page):

node cli.mjs example.com                         # human-readable report
node cli.mjs example.com --json                   # machine-readable
node cli.mjs example.com --fail-if-blocked GPTBot,ClaudeBot
node cli.mjs example.com --fail-if-allowed Bytespider
node cli.mjs --robots-file ./robots.txt --llms-file ./llms.txt

Get cli.mjs — Node 18+, nothing to install. It only talks to the domain you name (or reads local files) and makes no other network call.

Common questions

Which AI crawlers does this check?

A curated list of 20 named user-agent tokens publicly documented by their operators: GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI); ClaudeBot, Claude-User and Claude-SearchBot (Anthropic); Google-Extended (Google); Applebot-Extended (Apple); CCBot (Common Crawl); PerplexityBot and Perplexity-User (Perplexity); Bytespider (ByteDance); Amazonbot (Amazon); Meta-ExternalAgent, Meta-ExternalFetcher and FacebookBot (Meta); DuckAssistBot (DuckDuckGo); MistralAI-User (Mistral AI); Google-CloudVertexBot (Google); and Diffbot. It's a snapshot, not a live feed — operators add crawlers and change stated purposes over time, so verify anything load-bearing directly with the operator.

Does blocking a crawler here mean it actually obeys robots.txt?

No. robots.txt is a request, not an enforcement mechanism, and this tool only reports what your file says. Reputable operators state that they honor it for the tokens listed here, but nothing stops a request that lies about its own user-agent, and robots.txt cannot verify who is really asking.

What does "partially blocked" mean?

The crawler is allowed on some of the paths checked and blocked on others — most often because it falls through to your wildcard (User-agent: *) group, which usually blocks more than the root path.

Is my robots.txt uploaded anywhere?

No. The browser page parses the text you paste entirely in your own browser and never sends it anywhere. The CLI fetches only the domain you name, or reads a local file.

What is llms.txt?

An emerging, optional convention (see llmstxt.org) for a plain-text file at /llms.txt that gives an AI assistant a short, structured map of a site's key pages. This tool checks the basic shape the convention describes — an H1 title, an optional summary, and Markdown link sections — not full spec conformance.

More: exact robots.txt lines to block or allow each AI crawler, and what llms.txt is for.

Already using (or considering) a hosted, account-based service for this? See this tool vs. Known Agents (formerly Dark Visitors).

Usually check each crawler operator's own docs page by hand instead? See this tool vs. checking each crawler's docs page by hand.

Don't have an llms.txt yet, or want one built from your docs instead of hand-written? llms.txt Generator builds a spec-compliant one from a pasted page list, a sitemap.xml, or a local Markdown docs tree — the complementary job to this page's structural check.