The robots.txt decision nobody is making deliberately

The wrong question

The question people ask is "should I block AI bots?". It produces bad policy because it groups together crawlers that do completely different things and return completely different value.

The right question is per crawler, and it is an economic one: can this crawler return a visit or a citation in exchange for the bandwidth it consumes?

That splits the population three ways.

Three classes

Search crawlers index you and send clicks. Googlebot, Bingbot, DuckDuckBot, Applebot. Blocking any of these is a decision not to exist in that surface, and there is no upside.

Citation crawlers fetch a page to answer a live query and name the source. OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-User. These are what make answer engine visibility possible at all.

Training crawlers build a model. No link, no click, no attribution. GPTBot, Google-Extended, Applebot-Extended, anthropic-ai, CCBot, Bytespider.

Why the split is actionable

Because the major operators use separate user agents for the two jobs.

• Blocking GPTBot does not remove you from ChatGPT's search index, because that index is fetched by OAI-SearchBot.
• Blocking Google-Extended does not affect Google Search, which is Googlebot.
• Blocking Applebot-Extended does not affect Siri, which is Applebot.

That asymmetry is the entire mechanism. You can take the citations without donating the training corpus, and most sites do not know the option exists because the blog post they copied their blocklist from predates the split.

The trap that catches everyone once

A Disallow under User-agent: * does not apply to an agent that has its own named block. Named agents inherit nothing.

So this is wrong:

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Allow: /

Googlebot is now permitted in /admin/, because its block has no disallow list. The correct form repeats the list under every named agent:

User-agent: *
Allow: /
Disallow: /admin/

# Google: Search index. Also feeds AI Overviews.
User-agent: Googlebot
Allow: /
Disallow: /admin/

Generating the file from your route constants rather than maintaining it by hand is what stops this, and it costs about twenty lines.

What the numbers looked like

Over a 90 day window on a site of roughly 2,100 indexable URLs:

• Denied training crawlers: 31 percent of total bot requests, zero referral sessions.
• Allowed citation crawlers: 4 percent of total bot requests, measurable referral traffic from two engines.

That ratio is the argument. It is not that training crawlers are unethical, it is that they are a cost line with no matching revenue line, and a cost with no return is hard to defend on a solo product.

The honest caveats

robots.txt is a request, not an enforcement mechanism. Several crawlers, Bytespider most prominently, are widely reported to ignore it. If you need enforcement, that is a WAF rule at the edge, not a text file.

And there is a real argument for allowing training crawlers: models trained on your content may describe your product more accurately, and presence in a model's parametric knowledge has some value even if nobody can measure it. That argument is not silly. It is just unmeasurable, and I would rather make the decision explicitly than inherit it from a framework default.

What to do

Read your current robots.txt. If every named agent in it is not there for a stated reason, you have not made this decision yet, you have inherited it.

The bot economics chapter is free and goes crawler by crawler. The robots.txt generator builds the file with the reasoning as comments, because people read this file as part of evaluating a site.