#robots.txt

The robots.txt decision nobody is making deliberately

Most sites ship whatever robots.txt their framework generated. Here is the crawler-by-crawler analysis, and why the AI-versus-search framing produces the wrong policy.