Why does robots.txt is generated from code, not maintained by hand matter?

A hand-maintained robots.txt drifts from the routes it is supposed to describe. Generating it means adding a private route section cannot forget to disallow it. Verify it: Confirm no static robots.txt exists in the public directory and that `curl -s https://example.com/robots.txt` returns generated output.

Why does the crawler policy is a deliberate decision, split by whether a bot returns traffic matter?

The useful split is not AI versus search. It is whether a crawler can send a visit or a citation in exchange for the bandwidth. Blocking a training crawler does not remove you from that company's answer engine when the citation crawler is a separate user agent. Verify it: Run `curl -s https://example.com/robots.txt | grep -E 'GPTBot|OAI-SearchBot|ClaudeBot|Google-Extended'` and confirm each named agent has an explicit, intentional rule.

Why does aPI, auth and account routes are disallowed for every allowed agent matter?

A disallow under `User-agent: *` does not apply to an agent that has its own named block. Named agents inherit nothing. Verify it: Run `curl -s https://example.com/robots.txt` and confirm the private path list repeats under every named allowed agent, not only under the wildcard.

Why does robots.txt declares the sitemap and the canonical host matter?

The Sitemap directive is how a crawler finds the sitemap without you submitting it anywhere. Host is read by Bing and ignored by Google, which costs nothing. Verify it: Run `curl -s https://example.com/robots.txt | grep -i '^sitemap:'` and confirm an absolute URL.

Why does the sitemap is composed from the same data the pages render from matter?

A hardcoded URL list in a sitemap goes stale the first time a route is added. Composing from the content modules makes the sitemap a projection of reality rather than a claim about it. Verify it: Add a page, rebuild, and confirm it appears without editing the sitemap: `curl -s https://example.com/sitemap.xml | grep -c ' '`.

Why does lastModified comes from content dates, never from the build timestamp matter?

A sitemap claiming 2,100 pages changed at 03:14 this morning teaches the crawler the field is noise. It then stops reading it, and you lose the signal permanently. Verify it: Run `curl -s https://example.com/sitemap.xml | grep -o ' [^<]*' | cut -c10-19 | sort | uniq -c | sort -rn | head` and confirm no single date covers most of the URLs.

Why does a differentiated priority ladder, not 0.8 on everything matter?

Priority is a relative hint inside one site. Its only real job is telling a crawler with a limited budget which URLs to revisit first, and a flat value communicates nothing. Verify it: Run `curl -s https://example.com/sitemap.xml | grep -o ' [^<]*' | sort | uniq -c` and confirm at least four distinct values.

Why does hreflang alternates are emitted per URL in the sitemap matter?

Sitemap-level hreflang scales better than tag-level for a large site, and it is the only place to declare it for non-HTML resources. Verify it: Run `curl -s https://example.com/sitemap.xml | grep -c 'xhtml:link'` and confirm a non-zero count.

Crawling and indexing

Crawling and indexing covers 15 of the SEO for Solos checklist's implementation checks. robots.txt, sitemaps, feeds and push submission. Controls what gets fetched and how often.

Checks in group

High impact

impact 4 or 5

2.9

Average impact

of 5

Position

of 12 groups

0 of 15

Saved in this browser

Sync across devices

Crawling and indexing

robots.txt, sitemaps, feeds and push submission. Controls what gets fetched and how often.

robots.txt is generated from code, not maintained by hand3/S
Why it matters
A hand-maintained robots.txt drifts from the routes it is supposed to describe. Generating it means adding a private route section cannot forget to disallow it.
How to verify
Confirm no static robots.txt exists in the public directory and that `curl -s https://example.com/robots.txt` returns generated output.
crawl-robots-generatedImpact 3/5SmallRead the chapter
The crawler policy is a deliberate decision, split by whether a bot returns traffic4/M
Why it matters
The useful split is not AI versus search. It is whether a crawler can send a visit or a citation in exchange for the bandwidth. Blocking a training crawler does not remove you from that company's answer engine when the citation crawler is a separate user agent.
How to verify
curl -s https://example.com/robots.txt | grep -E 'GPTBot|OAI-SearchBot|ClaudeBot|Google-Extended'
Then confirm each named agent has an explicit, intentional rule.
crawl-bot-economicsImpact 4/5MediumRead the chapter
API, auth and account routes are disallowed for every allowed agent4/S
Why it matters
A disallow under `User-agent: *` does not apply to an agent that has its own named block. Named agents inherit nothing.
How to verify
curl -s https://example.com/robots.txt
Then confirm the private path list repeats under every named allowed agent, not only under the wildcard.
crawl-private-routes-disallowedImpact 4/5SmallRead the chapter
robots.txt declares the sitemap and the canonical host3/S
Why it matters
The Sitemap directive is how a crawler finds the sitemap without you submitting it anywhere. Host is read by Bing and ignored by Google, which costs nothing.
How to verify
curl -s https://example.com/robots.txt | grep -i '^sitemap:'
Then confirm an absolute URL.
crawl-sitemap-declaredImpact 3/5SmallRead the chapter
The sitemap is composed from the same data the pages render from4/M
Why it matters
A hardcoded URL list in a sitemap goes stale the first time a route is added. Composing from the content modules makes the sitemap a projection of reality rather than a claim about it.
How to verify
Add a page, rebuild, and confirm it appears without editing the sitemap: `curl -s https://example.com/sitemap.xml | grep -c '<loc>'`.
crawl-sitemap-generatedImpact 4/5MediumRead the chapter
lastModified comes from content dates, never from the build timestamp4/M
Why it matters
A sitemap claiming 2,100 pages changed at 03:14 this morning teaches the crawler the field is noise. It then stops reading it, and you lose the signal permanently.
How to verify
curl -s https://example.com/sitemap.xml | grep -o '<lastmod>[^<]*' | cut -c10-19 | sort | uniq -c | sort -rn | head
Then confirm no single date covers most of the URLs.
crawl-lastmodified-differentiatedImpact 4/5MediumRead the chapter
A differentiated priority ladder, not 0.8 on everything2/S
Why it matters
Priority is a relative hint inside one site. Its only real job is telling a crawler with a limited budget which URLs to revisit first, and a flat value communicates nothing.
How to verify
curl -s https://example.com/sitemap.xml | grep -o '<priority>[^<]*' | sort | uniq -c
Then confirm at least four distinct values.
crawl-priority-ladderImpact 2/5SmallRead the chapter
hreflang alternates are emitted per URL in the sitemap2/S
Why it matters
Sitemap-level hreflang scales better than tag-level for a large site, and it is the only place to declare it for non-HTML resources.
How to verify
curl -s https://example.com/sitemap.xml | grep -c 'xhtml:link'
Then confirm a non-zero count.
crawl-hreflangImpact 2/5SmallRead the chapter
A build-time test fails on duplicate, off-origin or trailing-slash sitemap URLs3/M
Why it matters
A sitemap containing a URL that redirects, 404s, or points at another host wastes crawl budget and lowers trust in the whole file. It is trivially caught by a test and almost never tested.
How to verify
Run the sitemap integrity suite: `npm run test -- sitemap`. It must fail the build, not warn.
crawl-sitemap-integrityImpact 3/5MediumRead the chapter
An RSS feed generated from the same source as the sitemap2/S
Why it matters
Feeds still drive real distribution through readers, aggregators and newsletter tools. Generating from one source means the two cannot disagree about what exists.
How to verify
curl -s https://example.com/feed.xml | head -5
Then validate at validator.w3.org/feed. Zero errors, not just zero fatal errors.
crawl-feed-validImpact 2/5SmallRead the chapter
Every interpolated value in the feed is XML-escaped2/S
Why it matters
One unescaped ampersand in a title invalidates the whole document, and most readers fail closed rather than skipping the item.
How to verify
Publish a post with `&`, `<` and `"` in the title, then run `curl -s https://example.com/feed.xml | xmllint --noout -` and confirm no error.
crawl-feed-escapedImpact 2/5SmallRead the chapter
The feed is declared in the head and linked from the footer1/S
Why it matters
An undeclared feed is only found by people who guess the URL. The alternate link is what feed readers auto-discover.
How to verify
curl -s https://example.com/ | grep 'application/rss+xml'
Then confirm a link element with rel=alternate.
crawl-feed-declaredImpact 1/5SmallRead the chapter
IndexNow batch submission on a schedule2/M
Why it matters
It is the only push mechanism that exists. Everything else is polling. Bing, Yandex, Naver and Seznam participate; Google does not.
How to verify
Trigger the cron route with a valid token and confirm a 200 or 202 from api.indexnow.org in the response body.
crawl-indexnowImpact 2/5MediumRead the chapter
The IndexNow key file is served through a constrained rewrite, not committed3/S
Why it matters
A key committed to a public repo lets anyone submit URLs on your host. Constraining the rewrite to a 32-hex-character pattern also stops the route being used to probe for arbitrary files.
How to verify
curl -s -o /dev/null -w '%{http_code}\n' https://example.com/not-a-key.txt
Then confirm 404, then confirm the real key path returns 200.
crawl-indexnow-key-protectedImpact 3/5SmallRead the chapter
No indexable URL is reachable only from the sitemap4/M
Why it matters
A page with no internal links pointing at it gets crawled rarely and ranks poorly, whatever the sitemap says. The sitemap is a hint; links are the graph.
How to verify
Crawl the site from the home page and diff the discovered URL set against the sitemap set. Anything in the sitemap and not in the crawl is orphaned.
crawl-orphan-pagesImpact 4/5MediumRead the chapter