The obvious way to check your robots.txt is to open it in a browser tab and read it. That works for what it checks: is the file syntactically sane, does it say what you meant. It does not answer the question that actually determines whether your sitemap gets read, because the file most people open by habit — the one at their own site's address — is usually not the file a crawler fetches at all.
Opening robots.txt and reading it is free and needs no tool: paste the URL, look at the text. It correctly answers "is this file well-formed" and "does this rule say what I think it says", for whichever file you happened to open.
This checker starts from your site's URL rather than a robots.txt URL, works out which file actually governs it under RFC 9309 (always the origin root, never a path prefix), and reports the gap when the two differ. Paste the origin-root file too, and it applies the longest-match-wins rule for you rather than leaving it to be eyeballed.
| Question | Where to answer it |
|---|---|
| Is the robots.txt I publish syntactically correct? | Either one. Opening it and reading it answers this directly, and so does this checker's inert-directive report. |
| Is the file I just read the one a crawler actually fetches? | This checker. The file at your own site's address is the governing file only when your site is at an origin root — otherwise it is a different URL you have to know to check. |
| Does a rule in the origin-root file block my path outright? | This checker, applying RFC 9309's longest-match rule to whatever you paste, rather than reading it by eye. |
| Will my Sitemap: line ever be auto-discovered? | This checker's core question. A file that looks correct when opened can still be a file nothing requests. |
| Is anything I paste uploaded anywhere? | Neither. Opening a URL involves no third party beyond the site itself, and this checker makes no network calls of its own. |
It confirms the file you publish is syntactically correct, which is a real thing to check. It does not confirm that file is the one a crawler reads. The URL you probably opened is your site's own robots.txt under its path prefix — exactly the file this tool exists to point out gets fetched by nothing.
Work out your site's origin root from the URL, open robots.txt there instead of at your own path, and read it for a Disallow rule that matches your path prefix — applying RFC 9309's longest-match-wins rule, where Allow only wins a tie. That is a different file, a different question, and a rule most people have not memorized, which is exactly why a step of it gets skipped.
It can, if you know to check that specific file and apply the longest-match rule correctly. It is easy to get backwards: a shorter Disallow and a longer, more specific Allow both matching your path resolve to Allow, not to whichever rule is read first or looks more prominent.
Yes — whether the file reads naturally, whether a comment explains an unusual rule, anything about tone or intent. This tool checks one mechanical question: which file governs, and what follows for sitemap discovery. It is not a substitute for reading the file that does apply.
Opening a URL in a browser tab involves no third party beyond the site itself. This checker runs entirely in your browser too and makes no network calls of its own — not even to fetch the robots.txt files it reports on.