Robots.txt checker

What robots.txt is, and is not

robots.txt is a plain-text file at a domain's root that asks well-behaved crawlers to stay out of certain paths, or grants specific crawlers access others do not have. It is entirely voluntary - nothing enforces it. A crawler that ignores it faces no technical barrier at all, which is why it must never be relied on to keep something actually private or secure; that needs authentication, not a request politely written in a text file.

Reading the groups

Rules are grouped under whichever User-agent: line precedes them - a specific crawler name (like Googlebot) or * for everyone not named more specifically elsewhere. A crawler follows the most specific group that names it, falling back to * if none does.

Allow, Disallow and crawl-delay

Disallow paths are off-limits to that group; Allow carves out exceptions within a disallowed area (useful when most of a directory is blocked but one file inside it should stay crawlable). Crawl-delay asks for a minimum gap between requests - respected by some crawlers, ignored by others (Google, notably, does not honor it and expects crawl rate to be set in Search Console instead).

The Sitemap declaration

A Sitemap: line can appear anywhere in the file and is how a site tells crawlers where its sitemap lives, independent of any user-agent group - check what it points to with the sitemap checker.