Why does robots.txt declares the sitemap and the canonical host matter?

The Sitemap directive is how a crawler finds the sitemap without you submitting it anywhere. Host is read by Bing and ignored by Google, which costs nothing. Verify it: Run `curl -s https://example.com/robots.txt | grep -i '^sitemap:'` and confirm an absolute URL.

Why does lastModified comes from content dates, never from the build timestamp matter?

A sitemap claiming 2,100 pages changed at 03:14 this morning teaches the crawler the field is noise. It then stops reading it, and you lose the signal permanently. Verify it: Run `curl -s https://example.com/sitemap.xml | grep -o ' [^<]*' | cut -c10-19 | sort | uniq -c | sort -rn | head` and confirm no single date covers most of the URLs.

Why does a differentiated priority ladder, not 0.8 on everything matter?

Priority is a relative hint inside one site. Its only real job is telling a crawler with a limited budget which URLs to revisit first, and a flat value communicates nothing. Verify it: Run `curl -s https://example.com/sitemap.xml | grep -o ' [^<]*' | sort | uniq -c` and confirm at least four distinct values.

Why does hreflang alternates are emitted per URL in the sitemap matter?

Sitemap-level hreflang scales better than tag-level for a large site, and it is the only place to declare it for non-HTML resources. Verify it: Run `curl -s https://example.com/sitemap.xml | grep -c 'xhtml:link'` and confirm a non-zero count.

Crawling and indexing

Crawling and indexing covers 15 of the SEO for Soloschecklist's implementation checks. robots.txt, sitemaps, feeds and push submission. Controls what gets fetched and how often.

Checks in group

Free snippets

2.9

Average impact

of 5

Position

of 12 groups

0 of 15

Saved in this browser

Sync across devices

Crawling and indexing

robots.txt, sitemaps, feeds and push submission. Controls what gets fetched and how often.

robots.txt is generated from code, not maintained by hand3/S

Why it matters

A hand-maintained robots.txt drifts from the routes it is supposed to describe. Generating it means adding a private route section cannot forget to disallow it.

How to verify

Confirm no static robots.txt exists in the public directory and that `curl -s https://example.com/robots.txt` returns generated output.

Implementation

kit/seo/robots.ts

/**
 * Code Kit: bot-economics robots policy.
 * Implements checks: crawl-robots-generated, crawl-bot-economics,
 *                    crawl-private-routes-disallowed, crawl-sitemap-declared
 *
 * The decision nobody makes deliberately: which crawlers get your content.
 * The useful split is not "AI bots" versus "search bots". It is whether a
 * crawler can send you traffic or a citation in return for the bandwidth.
 *
 *   Search crawlers      index you and send clicks.            Allow.
 *   Citation crawlers    fetch a page to answer a live query
 *                        and name the source.                  Allow.
 *   Training crawlers    build a model. No link, no click,
 *                        no attribution.                       Your call.
 *
 * Blocking a training crawler does not remove you from that company's answer
 * engine if its citation crawler is a separate user agent, which for OpenAI,
 * Anthropic, Perplexity and Google it currently is. That asymmetry is the whole
 * mechanism.
 */

export type CrawlerClass = "search" | "citation" | "training" | "seo-tool" | "archive";

export type CrawlerProfile = {
  userAgent: string;
  operator: string;
  class: CrawlerClass;
  /** Does allowing this crawler produce visits or citations? */
  sendsReferralTraffic: boolean;
  purpose: string;
  documentationUrl?: string;
};

/**
 * Crawler registry.
 *
 * Verified against each operator's published documentation. Recheck quarterly:
 * user agent names change, and new ones appear faster than anyone updates
 * robots.txt.
 */
export const CRAWLERS: CrawlerProfile[] = [
  {
    userAgent: "Googlebot",
    operator: "Google",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Google Search index. Also feeds AI Overviews.",
    documentationUrl: "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers",
  },
  {
    userAgent: "Bingbot",
    operator: "Microsoft",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Bing index. Also the retrieval layer behind Copilot.",
    documentationUrl: "https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0",
  },
  {
    userAgent: "DuckDuckBot",
    operator: "DuckDuckGo",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "DuckDuckGo index.",
  },
  {
    userAgent: "Applebot",
    operator: "Apple",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Siri and Spotlight suggestions.",
    documentationUrl: "https://support.apple.com/en-us/119829",
  },
  {
    userAgent: "OAI-SearchBot",
    operator: "OpenAI",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "ChatGPT search index. Links back to the source.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "ChatGPT-User",
    operator: "OpenAI",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Live fetch when a user asks ChatGPT about a specific URL.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "PerplexityBot",
    operator: "Perplexity",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Perplexity answer index. Cites sources inline.",
    documentationUrl: "https://docs.perplexity.ai/guides/bots",
  },
  {
    userAgent: "ClaudeBot",
    operator: "Anthropic",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Claude retrieval and citation.",
    documentationUrl: "https://support.anthropic.com/en/articles/8896518",
  },
  {
    userAgent: "Claude-User",
    operator: "Anthropic",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Live fetch when a Claude user asks about a specific URL.",
  },
  {
    userAgent: "Applebot-Extended",
    operator: "Apple",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Apple foundation model training. Opting out does not affect Siri.",
    documentationUrl: "https://support.apple.com/en-us/119829",
  },
  {
    userAgent: "GPTBot",
    operator: "OpenAI",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Model training corpus. Separate from OAI-SearchBot.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "Google-Extended",
    operator: "Google",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Gemini training. Blocking it does not remove you from Search.",
    documentationUrl: "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers",
  },
  {
    userAgent: "anthropic-ai",
    operator: "Anthropic",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Legacy training crawler identifier.",
  },
  {
    userAgent: "CCBot",
    operator: "Common Crawl",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Open crawl corpus used as training data by many labs.",
    documentationUrl: "https://commoncrawl.org/ccbot",
  },
  {
    userAgent: "Bytespider",
    operator: "ByteDance",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Training corpus. Widely reported to ignore robots.txt.",
  },
  {
    userAgent: "AhrefsBot",
    operator: "Ahrefs",
    class: "seo-tool",
    sendsReferralTraffic: false,
    purpose: "Backlink index. Allowing it makes your own backlink data visible to you and to competitors.",
  },
  {
    userAgent: "SemrushBot",
    operator: "Semrush",
    class: "seo-tool",
    sendsReferralTraffic: false,
    purpose: "Competitive index. Same trade as AhrefsBot.",
  },
  {
    userAgent: "ia_archiver",
    operator: "Internet Archive",
    class: "archive",
    sendsReferralTraffic: false,
    purpose: "Wayback Machine preservation.",
  },
];

export type RobotsPolicy = {
  /** Crawler classes permitted to crawl. */
  allowClasses: CrawlerClass[];
  /** Paths every permitted crawler is denied. */
  disallowPaths: string[];
  /** Absolute sitemap URLs. */
  sitemaps: string[];
  /** Canonical host, no scheme. Bing reads this; Google ignores it. */
  host?: string;
  /** Extra user agents to deny outright, beyond the class rules. */
  denyUserAgents?: string[];
  /** Emit the reasoning as comments. Leave on: people read this file. */
  annotate?: boolean;
  crawlers?: CrawlerProfile[];
};

export const RECOMMENDED_POLICY: Pick<RobotsPolicy, "allowClasses"> = {
  // Allow anything that can send a visit or a citation. Deny training-only
  // crawlers and commercial SEO indexes, which take bandwidth and return nothing.
  allowClasses: ["search", "citation"],
};

export type RobotsRule = {
  userAgent: string | string[];
  allow?: string | string[];
  disallow?: string | string[];
};

/** Structured output, for `app/robots.ts` in Next.js. */
export function robotsRules(policy: RobotsPolicy): RobotsRule[] {
  const crawlers = policy.crawlers ?? CRAWLERS;
  const allowed = crawlers.filter((c) => policy.allowClasses.includes(c.class));
  const denied = crawlers.filter((c) => !policy.allowClasses.includes(c.class));
  const denyList = [
    ...denied.map((c) => c.userAgent),
    ...(policy.denyUserAgents ?? []),
  ];

  const rules: RobotsRule[] = [
    // The wildcard rule governs every crawler not named below, including ones
    // that do not exist yet.
    { userAgent: "*", allow: "/", disallow: policy.disallowPaths },
  ];

  for (const crawler of allowed) {
    rules.push({ userAgent: crawler.userAgent, allow: "/", disallow: policy.disallowPaths });
  }
  for (const userAgent of denyList) {
    rules.push({ userAgent, disallow: "/" });
  }

  return rules;
}

/** Plain text output, for a static robots.txt or the generator tool. */
export function robotsTxt(policy: RobotsPolicy): string {
  const crawlers = policy.crawlers ?? CRAWLERS;
  const annotate = policy.annotate ?? true;
  const lines: string[] = [];

  if (annotate) {
    lines.push(
      "# robots.txt",
      "#",
      "# Policy: allow any crawler that can return a visit or a citation.",
      "# Deny crawlers that consume bandwidth and return neither.",
      "#",
    );
  }

  const emit = (rule: RobotsRule, comment?: string) => {
    if (annotate && comment) lines.push(`# ${comment}`);
    const agents = Array.isArray(rule.userAgent) ? rule.userAgent : [rule.userAgent];
    for (const agent of agents) lines.push(`User-agent: ${agent}`);
    for (const allow of toArray(rule.allow)) lines.push(`Allow: ${allow}`);
    for (const disallow of toArray(rule.disallow)) lines.push(`Disallow: ${disallow}`);
    lines.push("");
  };

  emit(
    { userAgent: "*", allow: "/", disallow: policy.disallowPaths },
    "Default policy for every crawler not named below.",
  );

  for (const crawler of crawlers.filter((c) => policy.allowClasses.includes(c.class))) {
    emit(
      { userAgent: crawler.userAgent, allow: "/", disallow: policy.disallowPaths },
      `${crawler.operator}: ${crawler.purpose}`,
    );
  }

  const denied = crawlers.filter((c) => !policy.allowClasses.includes(c.class));
  for (const crawler of denied) {
    emit(
      { userAgent: crawler.userAgent, disallow: "/" },
      `${crawler.operator}: ${crawler.purpose} No referral traffic.`,
    );
  }
  for (const userAgent of policy.denyUserAgents ?? []) {
    emit({ userAgent, disallow: "/" });
  }

  for (const sitemap of policy.sitemaps) lines.push(`Sitemap: ${sitemap}`);
  if (policy.host) lines.push(`Host: ${policy.host}`);

  return `${lines.join("\n").replace(/\n{3,}/g, "\n\n").trimEnd()}\n`;
}

function toArray(value: string | string[] | undefined): string[] {
  if (!value) return [];
  return Array.isArray(value) ? value : [value];
}

crawl-robots-generatedImpact 3/5SmallFreeRead the chapter

The crawler policy is a deliberate decision, split by whether a bot returns traffic4/M

Why it matters

The useful split is not AI versus search. It is whether a crawler can send a visit or a citation in exchange for the bandwidth. Blocking a training crawler does not remove you from that company's answer engine when the citation crawler is a separate user agent.

How to verify

curl -s https://example.com/robots.txt | grep -E 'GPTBot|OAI-SearchBot|ClaudeBot|Google-Extended'

Then confirm each named agent has an explicit, intentional rule.

Implementation

kit/seo/robots.ts

/**
 * Code Kit: bot-economics robots policy.
 * Implements checks: crawl-robots-generated, crawl-bot-economics,
 *                    crawl-private-routes-disallowed, crawl-sitemap-declared
 *
 * The decision nobody makes deliberately: which crawlers get your content.
 * The useful split is not "AI bots" versus "search bots". It is whether a
 * crawler can send you traffic or a citation in return for the bandwidth.
 *
 *   Search crawlers      index you and send clicks.            Allow.
 *   Citation crawlers    fetch a page to answer a live query
 *                        and name the source.                  Allow.
 *   Training crawlers    build a model. No link, no click,
 *                        no attribution.                       Your call.
 *
 * Blocking a training crawler does not remove you from that company's answer
 * engine if its citation crawler is a separate user agent, which for OpenAI,
 * Anthropic, Perplexity and Google it currently is. That asymmetry is the whole
 * mechanism.
 */

export type CrawlerClass = "search" | "citation" | "training" | "seo-tool" | "archive";

export type CrawlerProfile = {
  userAgent: string;
  operator: string;
  class: CrawlerClass;
  /** Does allowing this crawler produce visits or citations? */
  sendsReferralTraffic: boolean;
  purpose: string;
  documentationUrl?: string;
};

/**
 * Crawler registry.
 *
 * Verified against each operator's published documentation. Recheck quarterly:
 * user agent names change, and new ones appear faster than anyone updates
 * robots.txt.
 */
export const CRAWLERS: CrawlerProfile[] = [
  {
    userAgent: "Googlebot",
    operator: "Google",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Google Search index. Also feeds AI Overviews.",
    documentationUrl: "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers",
  },
  {
    userAgent: "Bingbot",
    operator: "Microsoft",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Bing index. Also the retrieval layer behind Copilot.",
    documentationUrl: "https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0",
  },
  {
    userAgent: "DuckDuckBot",
    operator: "DuckDuckGo",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "DuckDuckGo index.",
  },
  {
    userAgent: "Applebot",
    operator: "Apple",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Siri and Spotlight suggestions.",
    documentationUrl: "https://support.apple.com/en-us/119829",
  },
  {
    userAgent: "OAI-SearchBot",
    operator: "OpenAI",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "ChatGPT search index. Links back to the source.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "ChatGPT-User",
    operator: "OpenAI",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Live fetch when a user asks ChatGPT about a specific URL.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "PerplexityBot",
    operator: "Perplexity",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Perplexity answer index. Cites sources inline.",
    documentationUrl: "https://docs.perplexity.ai/guides/bots",
  },
  {
    userAgent: "ClaudeBot",
    operator: "Anthropic",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Claude retrieval and citation.",
    documentationUrl: "https://support.anthropic.com/en/articles/8896518",
  },
  {
    userAgent: "Claude-User",
    operator: "Anthropic",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Live fetch when a Claude user asks about a specific URL.",
  },
  {
    userAgent: "Applebot-Extended",
    operator: "Apple",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Apple foundation model training. Opting out does not affect Siri.",
    documentationUrl: "https://support.apple.com/en-us/119829",
  },
  {
    userAgent: "GPTBot",
    operator: "OpenAI",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Model training corpus. Separate from OAI-SearchBot.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "Google-Extended",
    operator: "Google",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Gemini training. Blocking it does not remove you from Search.",
    documentationUrl: "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers",
  },
  {
    userAgent: "anthropic-ai",
    operator: "Anthropic",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Legacy training crawler identifier.",
  },
  {
    userAgent: "CCBot",
    operator: "Common Crawl",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Open crawl corpus used as training data by many labs.",
    documentationUrl: "https://commoncrawl.org/ccbot",
  },
  {
    userAgent: "Bytespider",
    operator: "ByteDance",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Training corpus. Widely reported to ignore robots.txt.",
  },
  {
    userAgent: "AhrefsBot",
    operator: "Ahrefs",
    class: "seo-tool",
    sendsReferralTraffic: false,
    purpose: "Backlink index. Allowing it makes your own backlink data visible to you and to competitors.",
  },
  {
    userAgent: "SemrushBot",
    operator: "Semrush",
    class: "seo-tool",
    sendsReferralTraffic: false,
    purpose: "Competitive index. Same trade as AhrefsBot.",
  },
  {
    userAgent: "ia_archiver",
    operator: "Internet Archive",
    class: "archive",
    sendsReferralTraffic: false,
    purpose: "Wayback Machine preservation.",
  },
];

export type RobotsPolicy = {
  /** Crawler classes permitted to crawl. */
  allowClasses: CrawlerClass[];
  /** Paths every permitted crawler is denied. */
  disallowPaths: string[];
  /** Absolute sitemap URLs. */
  sitemaps: string[];
  /** Canonical host, no scheme. Bing reads this; Google ignores it. */
  host?: string;
  /** Extra user agents to deny outright, beyond the class rules. */
  denyUserAgents?: string[];
  /** Emit the reasoning as comments. Leave on: people read this file. */
  annotate?: boolean;
  crawlers?: CrawlerProfile[];
};

export const RECOMMENDED_POLICY: Pick<RobotsPolicy, "allowClasses"> = {
  // Allow anything that can send a visit or a citation. Deny training-only
  // crawlers and commercial SEO indexes, which take bandwidth and return nothing.
  allowClasses: ["search", "citation"],
};

export type RobotsRule = {
  userAgent: string | string[];
  allow?: string | string[];
  disallow?: string | string[];
};

/** Structured output, for `app/robots.ts` in Next.js. */
export function robotsRules(policy: RobotsPolicy): RobotsRule[] {
  const crawlers = policy.crawlers ?? CRAWLERS;
  const allowed = crawlers.filter((c) => policy.allowClasses.includes(c.class));
  const denied = crawlers.filter((c) => !policy.allowClasses.includes(c.class));
  const denyList = [
    ...denied.map((c) => c.userAgent),
    ...(policy.denyUserAgents ?? []),
  ];

  const rules: RobotsRule[] = [
    // The wildcard rule governs every crawler not named below, including ones
    // that do not exist yet.
    { userAgent: "*", allow: "/", disallow: policy.disallowPaths },
  ];

  for (const crawler of allowed) {
    rules.push({ userAgent: crawler.userAgent, allow: "/", disallow: policy.disallowPaths });
  }
  for (const userAgent of denyList) {
    rules.push({ userAgent, disallow: "/" });
  }

  return rules;
}

/** Plain text output, for a static robots.txt or the generator tool. */
export function robotsTxt(policy: RobotsPolicy): string {
  const crawlers = policy.crawlers ?? CRAWLERS;
  const annotate = policy.annotate ?? true;
  const lines: string[] = [];

  if (annotate) {
    lines.push(
      "# robots.txt",
      "#",
      "# Policy: allow any crawler that can return a visit or a citation.",
      "# Deny crawlers that consume bandwidth and return neither.",
      "#",
    );
  }

  const emit = (rule: RobotsRule, comment?: string) => {
    if (annotate && comment) lines.push(`# ${comment}`);
    const agents = Array.isArray(rule.userAgent) ? rule.userAgent : [rule.userAgent];
    for (const agent of agents) lines.push(`User-agent: ${agent}`);
    for (const allow of toArray(rule.allow)) lines.push(`Allow: ${allow}`);
    for (const disallow of toArray(rule.disallow)) lines.push(`Disallow: ${disallow}`);
    lines.push("");
  };

  emit(
    { userAgent: "*", allow: "/", disallow: policy.disallowPaths },
    "Default policy for every crawler not named below.",
  );

  for (const crawler of crawlers.filter((c) => policy.allowClasses.includes(c.class))) {
    emit(
      { userAgent: crawler.userAgent, allow: "/", disallow: policy.disallowPaths },
      `${crawler.operator}: ${crawler.purpose}`,
    );
  }

  const denied = crawlers.filter((c) => !policy.allowClasses.includes(c.class));
  for (const crawler of denied) {
    emit(
      { userAgent: crawler.userAgent, disallow: "/" },
      `${crawler.operator}: ${crawler.purpose} No referral traffic.`,
    );
  }
  for (const userAgent of policy.denyUserAgents ?? []) {
    emit({ userAgent, disallow: "/" });
  }

  for (const sitemap of policy.sitemaps) lines.push(`Sitemap: ${sitemap}`);
  if (policy.host) lines.push(`Host: ${policy.host}`);

  return `${lines.join("\n").replace(/\n{3,}/g, "\n\n").trimEnd()}\n`;
}

function toArray(value: string | string[] | undefined): string[] {
  if (!value) return [];
  return Array.isArray(value) ? value : [value];
}

crawl-bot-economicsImpact 4/5MediumFreeRead the chapter

API, auth and account routes are disallowed for every allowed agent4/S

Why it matters

A disallow under `User-agent: *` does not apply to an agent that has its own named block. Named agents inherit nothing.

How to verify

curl -s https://example.com/robots.txt

Then confirm the private path list repeats under every named allowed agent, not only under the wildcard.

Implementation

kit/seo/robots.ts

/**
 * Code Kit: bot-economics robots policy.
 * Implements checks: crawl-robots-generated, crawl-bot-economics,
 *                    crawl-private-routes-disallowed, crawl-sitemap-declared
 *
 * The decision nobody makes deliberately: which crawlers get your content.
 * The useful split is not "AI bots" versus "search bots". It is whether a
 * crawler can send you traffic or a citation in return for the bandwidth.
 *
 *   Search crawlers      index you and send clicks.            Allow.
 *   Citation crawlers    fetch a page to answer a live query
 *                        and name the source.                  Allow.
 *   Training crawlers    build a model. No link, no click,
 *                        no attribution.                       Your call.
 *
 * Blocking a training crawler does not remove you from that company's answer
 * engine if its citation crawler is a separate user agent, which for OpenAI,
 * Anthropic, Perplexity and Google it currently is. That asymmetry is the whole
 * mechanism.
 */

export type CrawlerClass = "search" | "citation" | "training" | "seo-tool" | "archive";

export type CrawlerProfile = {
  userAgent: string;
  operator: string;
  class: CrawlerClass;
  /** Does allowing this crawler produce visits or citations? */
  sendsReferralTraffic: boolean;
  purpose: string;
  documentationUrl?: string;
};

/**
 * Crawler registry.
 *
 * Verified against each operator's published documentation. Recheck quarterly:
 * user agent names change, and new ones appear faster than anyone updates
 * robots.txt.
 */
export const CRAWLERS: CrawlerProfile[] = [
  {
    userAgent: "Googlebot",
    operator: "Google",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Google Search index. Also feeds AI Overviews.",
    documentationUrl: "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers",
  },
  {
    userAgent: "Bingbot",
    operator: "Microsoft",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Bing index. Also the retrieval layer behind Copilot.",
    documentationUrl: "https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0",
  },
  {
    userAgent: "DuckDuckBot",
    operator: "DuckDuckGo",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "DuckDuckGo index.",
  },
  {
    userAgent: "Applebot",
    operator: "Apple",
    class: "search",
    sendsReferralTraffic: true,
    purpose: "Siri and Spotlight suggestions.",
    documentationUrl: "https://support.apple.com/en-us/119829",
  },
  {
    userAgent: "OAI-SearchBot",
    operator: "OpenAI",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "ChatGPT search index. Links back to the source.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "ChatGPT-User",
    operator: "OpenAI",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Live fetch when a user asks ChatGPT about a specific URL.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "PerplexityBot",
    operator: "Perplexity",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Perplexity answer index. Cites sources inline.",
    documentationUrl: "https://docs.perplexity.ai/guides/bots",
  },
  {
    userAgent: "ClaudeBot",
    operator: "Anthropic",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Claude retrieval and citation.",
    documentationUrl: "https://support.anthropic.com/en/articles/8896518",
  },
  {
    userAgent: "Claude-User",
    operator: "Anthropic",
    class: "citation",
    sendsReferralTraffic: true,
    purpose: "Live fetch when a Claude user asks about a specific URL.",
  },
  {
    userAgent: "Applebot-Extended",
    operator: "Apple",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Apple foundation model training. Opting out does not affect Siri.",
    documentationUrl: "https://support.apple.com/en-us/119829",
  },
  {
    userAgent: "GPTBot",
    operator: "OpenAI",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Model training corpus. Separate from OAI-SearchBot.",
    documentationUrl: "https://platform.openai.com/docs/bots",
  },
  {
    userAgent: "Google-Extended",
    operator: "Google",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Gemini training. Blocking it does not remove you from Search.",
    documentationUrl: "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers",
  },
  {
    userAgent: "anthropic-ai",
    operator: "Anthropic",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Legacy training crawler identifier.",
  },
  {
    userAgent: "CCBot",
    operator: "Common Crawl",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Open crawl corpus used as training data by many labs.",
    documentationUrl: "https://commoncrawl.org/ccbot",
  },
  {
    userAgent: "Bytespider",
    operator: "ByteDance",
    class: "training",
    sendsReferralTraffic: false,
    purpose: "Training corpus. Widely reported to ignore robots.txt.",
  },
  {
    userAgent: "AhrefsBot",
    operator: "Ahrefs",
    class: "seo-tool",
    sendsReferralTraffic: false,
    purpose: "Backlink index. Allowing it makes your own backlink data visible to you and to competitors.",
  },
  {
    userAgent: "SemrushBot",
    operator: "Semrush",
    class: "seo-tool",
    sendsReferralTraffic: false,
    purpose: "Competitive index. Same trade as AhrefsBot.",
  },
  {
    userAgent: "ia_archiver",
    operator: "Internet Archive",
    class: "archive",
    sendsReferralTraffic: false,
    purpose: "Wayback Machine preservation.",
  },
];

export type RobotsPolicy = {
  /** Crawler classes permitted to crawl. */
  allowClasses: CrawlerClass[];
  /** Paths every permitted crawler is denied. */
  disallowPaths: string[];
  /** Absolute sitemap URLs. */
  sitemaps: string[];
  /** Canonical host, no scheme. Bing reads this; Google ignores it. */
  host?: string;
  /** Extra user agents to deny outright, beyond the class rules. */
  denyUserAgents?: string[];
  /** Emit the reasoning as comments. Leave on: people read this file. */
  annotate?: boolean;
  crawlers?: CrawlerProfile[];
};

export const RECOMMENDED_POLICY: Pick<RobotsPolicy, "allowClasses"> = {
  // Allow anything that can send a visit or a citation. Deny training-only
  // crawlers and commercial SEO indexes, which take bandwidth and return nothing.
  allowClasses: ["search", "citation"],
};

export type RobotsRule = {
  userAgent: string | string[];
  allow?: string | string[];
  disallow?: string | string[];
};

/** Structured output, for `app/robots.ts` in Next.js. */
export function robotsRules(policy: RobotsPolicy): RobotsRule[] {
  const crawlers = policy.crawlers ?? CRAWLERS;
  const allowed = crawlers.filter((c) => policy.allowClasses.includes(c.class));
  const denied = crawlers.filter((c) => !policy.allowClasses.includes(c.class));
  const denyList = [
    ...denied.map((c) => c.userAgent),
    ...(policy.denyUserAgents ?? []),
  ];

  const rules: RobotsRule[] = [
    // The wildcard rule governs every crawler not named below, including ones
    // that do not exist yet.
    { userAgent: "*", allow: "/", disallow: policy.disallowPaths },
  ];

  for (const crawler of allowed) {
    rules.push({ userAgent: crawler.userAgent, allow: "/", disallow: policy.disallowPaths });
  }
  for (const userAgent of denyList) {
    rules.push({ userAgent, disallow: "/" });
  }

  return rules;
}

/** Plain text output, for a static robots.txt or the generator tool. */
export function robotsTxt(policy: RobotsPolicy): string {
  const crawlers = policy.crawlers ?? CRAWLERS;
  const annotate = policy.annotate ?? true;
  const lines: string[] = [];

  if (annotate) {
    lines.push(
      "# robots.txt",
      "#",
      "# Policy: allow any crawler that can return a visit or a citation.",
      "# Deny crawlers that consume bandwidth and return neither.",
      "#",
    );
  }

  const emit = (rule: RobotsRule, comment?: string) => {
    if (annotate && comment) lines.push(`# ${comment}`);
    const agents = Array.isArray(rule.userAgent) ? rule.userAgent : [rule.userAgent];
    for (const agent of agents) lines.push(`User-agent: ${agent}`);
    for (const allow of toArray(rule.allow)) lines.push(`Allow: ${allow}`);
    for (const disallow of toArray(rule.disallow)) lines.push(`Disallow: ${disallow}`);
    lines.push("");
  };

  emit(
    { userAgent: "*", allow: "/", disallow: policy.disallowPaths },
    "Default policy for every crawler not named below.",
  );

  for (const crawler of crawlers.filter((c) => policy.allowClasses.includes(c.class))) {
    emit(
      { userAgent: crawler.userAgent, allow: "/", disallow: policy.disallowPaths },
      `${crawler.operator}: ${crawler.purpose}`,
    );
  }

  const denied = crawlers.filter((c) => !policy.allowClasses.includes(c.class));
  for (const crawler of denied) {
    emit(
      { userAgent: crawler.userAgent, disallow: "/" },
      `${crawler.operator}: ${crawler.purpose} No referral traffic.`,
    );
  }
  for (const userAgent of policy.denyUserAgents ?? []) {
    emit({ userAgent, disallow: "/" });
  }

  for (const sitemap of policy.sitemaps) lines.push(`Sitemap: ${sitemap}`);
  if (policy.host) lines.push(`Host: ${policy.host}`);

  return `${lines.join("\n").replace(/\n{3,}/g, "\n\n").trimEnd()}\n`;
}

function toArray(value: string | string[] | undefined): string[] {
  if (!value) return [];
  return Array.isArray(value) ? value : [value];
}

crawl-private-routes-disallowedImpact 4/5SmallFreeRead the chapter

robots.txt declares the sitemap and the canonical host3/S
Why it matters
The Sitemap directive is how a crawler finds the sitemap without you submitting it anywhere. Host is read by Bing and ignored by Google, which costs nothing.
How to verify
curl -s https://example.com/robots.txt | grep -i '^sitemap:'
Then confirm an absolute URL.
Implementation snippet
The tested implementation from kit/seo/robots.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-sitemap-declaredImpact 3/5SmallRead the chapter

The sitemap is composed from the same data the pages render from4/M

Why it matters

A hardcoded URL list in a sitemap goes stale the first time a route is added. Composing from the content modules makes the sitemap a projection of reality rather than a claim about it.

How to verify

Add a page, rebuild, and confirm it appears without editing the sitemap: `curl -s https://example.com/sitemap.xml | grep -c '<loc>'`.

Implementation

kit/seo/sitemap.ts

/**
 * Code Kit: sitemap composer.
 * Implements checks: crawl-sitemap-generated, crawl-lastmodified-differentiated,
 *                    crawl-priority-ladder, crawl-hreflang, crawl-sitemap-integrity
 *
 * The failure this module exists to prevent: one build timestamp stamped on
 * every URL. A sitemap that claims 2,100 pages changed at 03:14 this morning is
 * worse than a sitemap with no lastModified at all, because the crawler learns
 * the field is noise and stops reading it. Every entry here carries a date that
 * came from the content.
 */

export type ChangeFrequency =
  | "always"
  | "hourly"
  | "daily"
  | "weekly"
  | "monthly"
  | "yearly"
  | "never";

export type SitemapEntry = {
  url: string;
  lastModified: Date;
  changeFrequency?: ChangeFrequency;
  priority?: number;
  alternates?: { languages: Record<string, string> };
};

export type RouteGroup<T> = {
  /** Group name, used in the integrity report. */
  name: string;
  items: T[];
  /** Absolute or site-relative URL for an item. */
  url: (item: T) => string;
  /** Real date from the content. Never `new Date()`. */
  lastModified: (item: T) => Date | string;
  priority: number;
  changeFrequency: ChangeFrequency;
};

/**
 * Priority ladder.
 *
 * Priority is a relative hint within one site, not a ranking input. Its only
 * real job is telling a crawler which URLs to revisit first when it has a
 * limited budget. Flat 0.8 on everything communicates nothing.
 */
export const PRIORITY = {
  home: 1.0,
  primaryConversion: 0.95,
  primaryContent: 0.9,
  hub: 0.85,
  programmatic: 0.8,
  editorial: 0.7,
  reference: 0.6,
  taxonomy: 0.5,
  legal: 0.3,
} as const;

export type SitemapOptions = {
  siteUrl: string;
  /** Alternate language codes to emit per URL. Include x-default. */
  languages?: string[];
};

export function buildSitemap(
  groups: Array<RouteGroup<never> | RouteGroup<unknown>>,
  opts: SitemapOptions,
): SitemapEntry[] {
  const origin = opts.siteUrl.replace(/\/+$/, "");
  const entries: SitemapEntry[] = [];
  const seen = new Set<string>();

  for (const group of groups as Array<RouteGroup<unknown>>) {
    for (const item of group.items) {
      const raw = group.url(item);
      const url = raw.startsWith("http") ? raw : `${origin}${raw.startsWith("/") ? "" : "/"}${raw}`;
      const normalised = url.replace(/(?<!:)\/{2,}/g, "/").replace(/\/$/, "") || origin;
      if (seen.has(normalised)) continue;
      seen.add(normalised);

      const lastModifiedRaw = group.lastModified(item);
      const lastModified =
        lastModifiedRaw instanceof Date ? lastModifiedRaw : new Date(lastModifiedRaw);

      entries.push({
        url: normalised,
        lastModified: Number.isNaN(lastModified.getTime()) ? new Date(0) : lastModified,
        changeFrequency: group.changeFrequency,
        priority: group.priority,
        ...(opts.languages?.length
          ? {
              alternates: {
                languages: Object.fromEntries(
                  opts.languages.map((lang) => [lang, normalised]),
                ),
              },
            }
          : {}),
      });
    }
  }

  return entries;
}

/**
 * Integrity report. Run this in a test so a sitemap regression fails the build
 * rather than quietly shipping.
 */
export function sitemapIntegrity(entries: SitemapEntry[], siteUrl: string) {
  const origin = siteUrl.replace(/\/+$/, "");
  const urls = entries.map((entry) => entry.url);
  const duplicates = urls.filter((url, index) => urls.indexOf(url) !== index);
  const offOrigin = urls.filter((url) => !url.startsWith(origin));
  const trailingSlash = urls.filter((url) => url !== origin && url.endsWith("/"));
  const epochDates = entries.filter((entry) => entry.lastModified.getTime() === 0);
  const futureDates = entries.filter((entry) => entry.lastModified.getTime() > Date.now() + 86_400_000);

  // If more than a quarter of URLs share one timestamp, lastModified is a build
  // stamp rather than a content date.
  const byTimestamp = new Map<number, number>();
  for (const entry of entries) {
    const day = Math.floor(entry.lastModified.getTime() / 86_400_000);
    byTimestamp.set(day, (byTimestamp.get(day) ?? 0) + 1);
  }
  const largestCluster = Math.max(0, ...byTimestamp.values());
  const clusteredRatio = entries.length > 0 ? largestCluster / entries.length : 0;

  return {
    total: entries.length,
    duplicates,
    offOrigin,
    trailingSlash,
    epochDates: epochDates.map((entry) => entry.url),
    futureDates: futureDates.map((entry) => entry.url),
    clusteredRatio,
    distinctDays: byTimestamp.size,
    ok:
      duplicates.length === 0 &&
      offOrigin.length === 0 &&
      trailingSlash.length === 0 &&
      epochDates.length === 0 &&
      futureDates.length === 0,
  };
}

/** Serialise to XML for a hand-rolled route or a static file. */
export function sitemapXml(entries: SitemapEntry[]): string {
  const urls = entries
    .map((entry) => {
      const alternates = entry.alternates?.languages
        ? Object.entries(entry.alternates.languages)
            .map(
              ([lang, href]) =>
                `    <xhtml:link rel="alternate" hreflang="${escapeXml(lang)}" href="${escapeXml(href)}" />`,
            )
            .join("\n")
        : "";
      return [
        "  <url>",
        `    <loc>${escapeXml(entry.url)}</loc>`,
        `    <lastmod>${entry.lastModified.toISOString()}</lastmod>`,
        entry.changeFrequency ? `    <changefreq>${entry.changeFrequency}</changefreq>` : "",
        entry.priority !== undefined ? `    <priority>${entry.priority.toFixed(2)}</priority>` : "",
        alternates,
        "  </url>",
      ]
        .filter(Boolean)
        .join("\n");
    })
    .join("\n");

  return [
    '<?xml version="1.0" encoding="UTF-8"?>',
    '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">',
    urls,
    "</urlset>",
    "",
  ].join("\n");
}

export function escapeXml(value: string): string {
  return value
    .replace(/&/g, "&amp;")
    .replace(/</g, "&lt;")
    .replace(/>/g, "&gt;")
    .replace(/"/g, "&quot;")
    .replace(/'/g, "&apos;");
}

crawl-sitemap-generatedImpact 4/5MediumFreeRead the chapter

lastModified comes from content dates, never from the build timestamp4/M
Why it matters
A sitemap claiming 2,100 pages changed at 03:14 this morning teaches the crawler the field is noise. It then stops reading it, and you lose the signal permanently.
How to verify
curl -s https://example.com/sitemap.xml | grep -o '<lastmod>[^<]*' | cut -c10-19 | sort | uniq -c | sort -rn | head
Then confirm no single date covers most of the URLs.
Implementation snippet
The tested implementation from kit/seo/sitemap.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-lastmodified-differentiatedImpact 4/5MediumRead the chapter
A differentiated priority ladder, not 0.8 on everything2/S
Why it matters
Priority is a relative hint inside one site. Its only real job is telling a crawler with a limited budget which URLs to revisit first, and a flat value communicates nothing.
How to verify
curl -s https://example.com/sitemap.xml | grep -o '<priority>[^<]*' | sort | uniq -c
Then confirm at least four distinct values.
Implementation snippet
The tested implementation from kit/seo/sitemap.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-priority-ladderImpact 2/5SmallRead the chapter
hreflang alternates are emitted per URL in the sitemap2/S
Why it matters
Sitemap-level hreflang scales better than tag-level for a large site, and it is the only place to declare it for non-HTML resources.
How to verify
curl -s https://example.com/sitemap.xml | grep -c 'xhtml:link'
Then confirm a non-zero count.
Implementation snippet
The tested implementation from kit/seo/sitemap.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-hreflangImpact 2/5SmallRead the chapter
A build-time test fails on duplicate, off-origin or trailing-slash sitemap URLs3/M
Why it matters
A sitemap containing a URL that redirects, 404s, or points at another host wastes crawl budget and lowers trust in the whole file. It is trivially caught by a test and almost never tested.
How to verify
Run the sitemap integrity suite: `npm run test -- sitemap`. It must fail the build, not warn.
Implementation snippet
The tested implementation from kit/seo/sitemap.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-sitemap-integrityImpact 3/5MediumRead the chapter
An RSS feed generated from the same source as the sitemap2/S
Why it matters
Feeds still drive real distribution through readers, aggregators and newsletter tools. Generating from one source means the two cannot disagree about what exists.
How to verify
curl -s https://example.com/feed.xml | head -5
Then validate at validator.w3.org/feed. Zero errors, not just zero fatal errors.
Implementation snippet
The tested implementation from kit/seo/feeds.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-feed-validImpact 2/5SmallRead the chapter
Every interpolated value in the feed is XML-escaped2/S
Why it matters
One unescaped ampersand in a title invalidates the whole document, and most readers fail closed rather than skipping the item.
How to verify
Publish a post with `&`, `<` and `"` in the title, then run `curl -s https://example.com/feed.xml | xmllint --noout -` and confirm no error.
Implementation snippet
The tested implementation from kit/seo/feeds.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-feed-escapedImpact 2/5SmallRead the chapter
The feed is declared in the head and linked from the footer1/S
Why it matters
An undeclared feed is only found by people who guess the URL. The alternate link is what feed readers auto-discover.
How to verify
curl -s https://example.com/ | grep 'application/rss+xml'
Then confirm a link element with rel=alternate.
Implementation snippet
The tested implementation from kit/seo/feeds.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-feed-declaredImpact 1/5SmallRead the chapter
IndexNow batch submission on a schedule2/M
Why it matters
It is the only push mechanism that exists. Everything else is polling. Bing, Yandex, Naver and Seznam participate; Google does not.
How to verify
Trigger the cron route with a valid token and confirm a 200 or 202 from api.indexnow.org in the response body.
Implementation snippet
The tested implementation from kit/seo/indexnow.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-indexnowImpact 2/5MediumRead the chapter
The IndexNow key file is served through a constrained rewrite, not committed3/S
Why it matters
A key committed to a public repo lets anyone submit URLs on your host. Constraining the rewrite to a 32-hex-character pattern also stops the route being used to probe for arbitrary files.
How to verify
curl -s -o /dev/null -w '%{http_code}\n' https://example.com/not-a-key.txt
Then confirm 404, then confirm the real key path returns 200.
Implementation snippet
The tested implementation from kit/seo/indexnow.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-indexnow-key-protectedImpact 3/5SmallRead the chapter
No indexable URL is reachable only from the sitemap4/M
Why it matters
A page with no internal links pointing at it gets crawled rarely and ranks poorly, whatever the sitemap says. The sitemap is a hint; links are the graph.
How to verify
Crawl the site from the home page and diff the discovered URL set against the sitemap set. Anything in the sitemap and not in the crawl is orphaned.
Implementation snippet
The tested implementation from kit/seo/sitemap.ts, plus the test that fails if it regresses.
Unlock with the Code Kit$14930 day refund, no questions.
crawl-orphan-pagesImpact 4/5MediumRead the chapter