SEO for Solos
Free checklist
Chapter 05Paid

Crawling and indexing

robots.txt generated from code, private routes that stay private, orphan detection, and IndexNow. Controls what gets fetched, how often, and by whom.

4 min read · updated 2026-07-26· last reviewed 2026-07-26

A hand-written robots.txt drifts from the routes it is supposed to describe. Somebody adds an admin area, ships it, and six weeks later notices the login page in the index. Generating the file from the same route constants the app uses means adding a private route cannot forget to disallow it.

PRIVATE_PATHS is the same array the middleware uses to decide what requires authentication. One list, two consumers, no drift.

The rest of "Crawling and indexing"

844 words covering 4 sections, with the implementation code for 6 checks. The table of contents below is the full outline, so you can see exactly what is behind this.

Already bought it? Read it in your library.

Unlock with the Playbook$14930 day refund, no questions.

What this chapter covers

  • robots.txt is generated, not maintained
    • The principle
    • The implementation
  • Private routes need two mechanisms, not one
    • The principle
  • Orphan pages
    • The principle
    • The implementation
  • IndexNow
    • The principle
    • The implementation

Checks this chapter covers

Each one has a command you can run against your own site.