Crawling and indexing

robots.txt is generated, not maintained

The principle

A hand-written robots.txt drifts from the routes it is supposed to describe. Somebody adds an admin area, ships it, and six weeks later notices the login page in the index. Generating the file from the same route constants the app uses means adding a private route cannot forget to disallow it.

The implementation

// app/robots.ts
import { robotsRules, RECOMMENDED_POLICY } from "@/kit/seo/robots";

export default function robots() {
  return {
    rules: robotsRules({
      ...RECOMMENDED_POLICY,
      disallowPaths: PRIVATE_PATHS,
      sitemaps: [`${SITE_URL}/sitemap.xml`],
      host: CANONICAL_HOST,
    }),
    sitemap: `${SITE_URL}/sitemap.xml`,
  };
}

PRIVATE_PATHS is the same array the middleware uses to decide what requires authentication. One list, two consumers, no drift.

Private routes need two mechanisms, not one

The principle

robots.txt stops the crawl. It does not stop the indexing. A URL that is disallowed but linked from somewhere public can still appear in results as a bare URL with no title and no snippet, because the crawler was never allowed to fetch the page and read its noindex.

The two mechanisms do different jobs and you need both: robots.txt to save crawl budget, noindex to guarantee removal. If a URL absolutely must not appear, allow the crawl and use the meta tag.

Orphan pages

The principle

A page reachable only from the sitemap is a page with no internal links pointing at it. It will be crawled rarely and rank poorly regardless of what the sitemap claims, because the sitemap is a hint and links are the graph.

This is the most under-diagnosed indexing problem on large sites, because the sitemap looks complete and the pages exist.

The implementation

Crawl from the home page, collect every internal URL, and diff against the sitemap:

const discovered = await crawlFrom(`${SITE_URL}/`);
const submitted = new Set(sitemapEntries.map((entry) => entry.url));
const orphans = [...submitted].filter((url) => !discovered.has(url));

The fix is structural: hub pages, sibling links inside programmatic tiers, and a footer that carries the site's important routes on every page. The Programmatic pages chapter covers the sibling linking pattern.

IndexNow

The principle

IndexNow is the only push mechanism that exists. Everything else is polling: you publish, and you wait for a crawler to come back. Bing, Yandex, Naver and Seznam participate. Google does not, and has said it does not plan to.

That makes it a small, cheap win rather than a large one. It costs one route and one scheduled job, and for a site whose content changes weekly it removes days of latency on the engines that do support it.

The implementation

export async function submitToIndexNow(opts: {
  siteUrl: string; key: string; urls: string[];
}): Promise<IndexNowResult> {
  const { host } = new URL(opts.siteUrl);
  const keyLocation = `${opts.siteUrl.replace(/\/+$/, "")}/${opts.key}.txt`;
  const unique = [...new Set(opts.urls)].filter((url) => url.startsWith(opts.siteUrl));
  const batches = chunk(unique, 10_000);
  // one POST per batch, host and keyLocation in the body
}

Two things go wrong in practice, and both are avoidable.

The key leaks. Anyone holding it can submit arbitrary URLs on your host. Serve the key file from an environment variable through a rewrite constrained to the exact key pattern, so the route cannot also be used to probe for other text files at the root:

{ source: "/:key([a-f0-9]{32}).txt", destination: "/api/indexnow-key/:key" }

The cron route is unauthenticated. Anyone can then trigger unbounded outbound requests from your infrastructure. Guard it with a timing-safe comparison and a deliberate delay on failure:

export function timingSafeEqual(a: string, b: string): boolean {
  const aBytes = new TextEncoder().encode(a);
  const bBytes = new TextEncoder().encode(b);
  let mismatch = aBytes.length === bBytes.length ? 0 : 1;
  const length = Math.max(aBytes.length, bBytes.length);
  for (let i = 0; i < length; i += 1) mismatch |= (aBytes[i] ?? 0) ^ (bBytes[i] ?? 0);
  return mismatch === 0;
}

A plain === returns as soon as two bytes differ, which leaks the length of the matching prefix to anyone who can measure response time across enough requests.