Sitemaps and feeds

One build timestamp on 2,100 URLs is worse than none

The principle

This is the highest-value idea in the chapter, so it goes first. A sitemap claiming that 2,100 pages all changed at 03:14 this morning is not a freshness signal. It is noise, and a crawler that encounters it a few times learns that your lastmod field carries no information and stops reading it.

You do not get that trust back by fixing it later. You are then a site with a correct lastmod that nobody reads.

Every entry's date has to come from the content it describes: the file's frontmatter date for a chapter, the published date for a post, the snapshot generation date for a programmatic tier, the last edit for a static page.

The implementation

export type RouteGroup<T> = {
  name: string;
  items: T[];
  url: (item: T) => string;
  lastModified: (item: T) => Date | string;   // real date, never new Date()
  priority: number;
  changeFrequency: ChangeFrequency;
};

Each group provides its own accessor. The composer never invents a date:

const groups = [
  { name: "playbook", items: getAllSections(),
    url: (s) => `/playbook/${s.slug}`,
    lastModified: (s) => s.updatedAt,
    priority: PRIORITY.primaryContent, changeFrequency: "monthly" },
  { name: "blog", items: getAllPosts(),
    url: (p) => `/blog/${p.slug}`,
    lastModified: (p) => p.updatedAt ?? p.publishedAt,
    priority: PRIORITY.editorial, changeFrequency: "monthly" },
];

Detecting the failure is a one-liner. If a single day covers most of the URLs, you have a build stamp:

curl -s https://example.com/sitemap.xml \
  | grep -o '<lastmod>[^<]*' | cut -c10-19 \
  | sort | uniq -c | sort -rn | head

The priority ladder

The principle

Priority is a relative hint inside one site. It is not a ranking input and no engine treats it as one. Its only real job is telling a crawler with a limited budget which of your URLs to revisit first.

Flat 0.8 on every URL communicates nothing, which is the same as omitting the field. A ladder with four or more distinct values communicates a hierarchy.

The implementation

export const PRIORITY = {
  home: 1.0,
  primaryConversion: 0.95,
  primaryContent: 0.9,
  hub: 0.85,
  programmatic: 0.8,
  editorial: 0.7,
  reference: 0.6,
  taxonomy: 0.5,
  legal: 0.3,
};

changeFrequency should match reality rather than aspiration. Declaring daily on a page that changes twice a year teaches the crawler the same lesson a build-stamped lastmod does.

The integrity test

The principle

A sitemap containing a URL that redirects, 404s, or points at another host wastes crawl budget and lowers trust in the whole file. Every one of those defects is mechanically detectable, and almost nobody tests for them.

The implementation

export function sitemapIntegrity(entries: SitemapEntry[], siteUrl: string) {
  const urls = entries.map((entry) => entry.url);
  return {
    duplicates: urls.filter((url, index) => urls.indexOf(url) !== index),
    offOrigin: urls.filter((url) => !url.startsWith(origin)),
    trailingSlash: urls.filter((url) => url !== origin && url.endsWith("/")),
    epochDates: entries.filter((entry) => entry.lastModified.getTime() === 0),
    futureDates: entries.filter((entry) => entry.lastModified.getTime() > Date.now() + 86_400_000),
    clusteredRatio,
  };
}

Run it as a test, not as a report. A warning in a log is a warning nobody reads.

Feeds

The principle

RSS still drives real distribution through readers, aggregators, newsletter tools and several AI ingestion pipelines. The reason to care here is narrower: generating the feed from the same entry list as the sitemap means the two cannot disagree about what exists.

The implementation

The failure mode is escaping. One unescaped ampersand in a title invalidates the whole document, and most readers fail closed rather than skipping the item:

export function escapeXml(value: string): string {
  return value
    .replace(/&/g, "&")
    .replace(/</g, "&lt;")
    .replace(/>/g, "&gt;")
    .replace(/"/g, "&quot;")
    .replace(/'/g, "&apos;");
}

And inside content:encoded, a literal ]]> in the body terminates the CDATA section early:

function stripCdataTerminator(html: string): string {
  return html.replace(/]]>/g, "]]>");
}

Declare the feed in the head and link it from the footer. An undeclared feed is only found by people who guess the URL.

alternates: {
  types: { "application/rss+xml": `${SITE_URL}/feed.xml` },
},