robots.txt
robots.txt is a plain text file at a site's root declaring which user agents may fetch which paths.
Why it matters
It is the only sitewide crawl control most sites have, and the crawler policy it encodes is usually a framework default rather than a decision.
How it is measured
Fetch /robots.txt and confirm each named agent has an intentional rule, including the private path list repeated under every named agent.
Checks that involve robots.txt
Related terms
Questions
What is robots.txt?
robots.txt is a plain text file at a site's root declaring which user agents may fetch which paths.
Why does robots.txt matter?
It is the only sitewide crawl control most sites have, and the crawler policy it encodes is usually a framework default rather than a decision.
How is robots.txt measured?
Fetch /robots.txt and confirm each named agent has an intentional rule, including the private path list repeated under every named agent.