Crawling

2 posts.

• Crawling

  Differentiated lastModified: why one build timestamp on 2,100 URLs is worse than none

  A sitemap claiming every page changed at 03:14 this morning is not a freshness signal. It is noise, and the trust you lose is not recoverable by fixing it later.

• Crawling

  The robots.txt decision nobody is making deliberately

  Most sites ship whatever robots.txt their framework generated. Here is the crawler-by-crawler analysis, and why the AI-versus-search framing produces the wrong policy.