Bot economics

The wrong question and the right one

The principle

The question people ask is "should I block AI bots?". It is the wrong question, because it groups together crawlers that do completely different things and returns completely different value.

The right question is per crawler: can this crawler return a visit or a citation in exchange for the bandwidth it consumes?

That splits the population three ways:

• Search crawlers index you and send clicks. Allow.
• Citation crawlers fetch a page to answer a live query and name the source. Allow.
• Training crawlers build a model. No link, no click, no attribution. Your call, and it is a genuine business decision rather than a technical one.

Why the split works

Because the major operators use separate user agents for the two jobs. Blocking OpenAI's training crawler does not remove you from ChatGPT's search index, because that index is fetched by a different agent. The same holds for Google and for Anthropic. That asymmetry is the entire mechanism, and it means you can take the citations without donating the training corpus.

The crawlers, one by one

Search crawlers: allow

Blocking any of these is a decision to not exist in that surface. There is no upside unless you are under crawl-rate pressure, and the fix for that is a crawl-delay conversation with the operator, not a block.

Citation crawlers: allow

These are the crawlers that make answer engine visibility possible. Blocking them is the AEO equivalent of blocking Googlebot, and a surprising number of sites do it by accident by using a blocklist copied from a blog post that predates the split.

Training crawlers: your decision

The argument for allowing them: models trained on your content may describe your product more accurately, and the effect on brand presence in a model's parametric knowledge is real if unmeasurable.

The argument against: it is a pure cost. Bandwidth out, nothing back, no attribution, no link.

There is no correct answer here, only a decision. The recommended default in this playbook is to deny, because a cost with no measurable return is hard to defend, and because the citation crawlers remain allowed so answer engine visibility is unaffected.

Commercial SEO crawlers: usually deny

AhrefsBot and SemrushBot index your backlinks. Allowing them makes your backlink profile visible to you through their tools, and equally visible to your competitors. It is a genuine trade rather than an obvious call. If you do not pay for either tool, you are paying bandwidth for a service you cannot use.

The implementation

export const RECOMMENDED_POLICY = {
  allowClasses: ["search", "citation"],
};

export function robotsTxt(policy: RobotsPolicy): string {
  // one block per allowed crawler with the full disallow list repeated,
  // one Disallow: / block per denied crawler,
  // then Sitemap and Host
}

Annotate the generated file. People read robots.txt as part of evaluating a site, and a file that explains its own reasoning is a small credibility signal that costs nothing:

# Default policy for every crawler not named below.
User-agent: *
Allow: /
Disallow: /api/

# OpenAI: ChatGPT search index. Links back to the source.
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/

# OpenAI: Model training corpus. Separate from OAI-SearchBot. No referral traffic.
User-agent: GPTBot
Disallow: /