SUBPATH ROBOTS CHECK · RUNS IN YOUR BROWSER

Is your sitemap ever actually read?

A crawler fetches robots.txt from exactly one place: the origin root. If your site lives under a path prefix — user.github.io/repo/, group.gitlab.io/project/, a CDN subdirectory — then the robots.txt you publish is at an address nothing ever requests, and the Sitemap: line inside it is never seen. Your pages are never enumerated. Nothing errors when this happens, which is why it usually goes unnoticed for years.

Nothing is uploaded. What you paste is read in this browser and never leaves your device, and this page makes no network calls — not even to fetch the robots.txt it tells you about.

1. Where is your site published?

Paste the address of your site's home page. A trailing index.html is fine — it is read as the directory it sits in.

Enter a URL above and click Check.

2. Optional: paste the robots.txt files

The check above works from the URL alone. These two make the report more specific, and the second one is the only way to find out whether a file you do not control is blocking you outright.

Add robots.txt contents

Reports which of its directives are inert.

Open the origin-root URL the report names, copy what is there, and paste it here. A 404 means no file exists, which crawlers read as no restrictions — leave this blank in that case.

Run it in CI instead

Zero dependencies, Node 18+. It opens no sockets at all unless you pass --fetch.

node cli.mjs check https://your-name.github.io/your-repo/
node cli.mjs check https://your-name.github.io/your-repo/ --fetch --fail-on-critical
node cli.mjs explain

--fail-on-critical exits 1 when the governing file is not yours, so a deploy pipeline can refuse to ship a site whose pages will never be enumerated.

Questions people actually ask

Why is my GitHub Pages sitemap never read?

Crawlers fetch robots.txt from one fixed place: the origin root. A project site at user.github.io/repo/ publishes its robots.txt at user.github.io/repo/robots.txt, which nothing ever requests, so the Sitemap: line inside it is never seen and the sitemap is never fetched.

Does this mean my pages are not indexed at all?

No. Crawling by following links is unaffected — any page linked from a page a crawler already knows can still be found and indexed. What breaks is enumeration: pages that only appear in the sitemap and are not linked from a crawled page are never discovered. That is why the failure is silent, since some pages do get indexed.

Whose robots.txt actually governs a GitHub Pages project site?

The one served by the repository named <user>.github.io — the account's user or organization site. If that repository does not exist, GitHub returns 404 for the file, which crawlers treat as no restrictions. Either way, a project site cannot put a Sitemap: line anywhere a crawler will read it.

Can I at least block a crawler from my subpath site?

Not with robots.txt. A Disallow: you write in a subpath robots.txt is as inert as the Sitemap: line. Use a per-page <meta name="robots" content="noindex"> tag or an X-Robots-Tag header instead — those need no origin-root file.

Does a clean result mean my site's SEO is fine?

No. This checks one specific, mechanical thing: whether the robots.txt a crawler reads is the one your site publishes, and what follows for sitemap discovery. It is not an SEO audit and says nothing about your content, your links, or whether any search engine has chosen to index you.

More on the mechanism, and the full list of what this cannot tell you: why a subpath site cannot govern its robots.txt, or how this compares to just opening robots.txt and reading it.