robots.txt generator
Build robots.txt from a crawler-by-crawler decision, with what each bot does and whether it returns referral traffic. Outputs both robots.txt and the equivalent app/robots.ts.
# robots.txt
#
# Policy: allow any crawler that can return a visit or a citation.
# Deny crawlers that consume bandwidth and return neither.
#
# Default policy for every crawler not named below.
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# Google: Google Search index. Also feeds AI Overviews.
User-agent: Googlebot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# Microsoft: Bing index. Also the retrieval layer behind Copilot.
User-agent: Bingbot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# DuckDuckGo: DuckDuckGo index.
User-agent: DuckDuckBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# Apple: Siri and Spotlight suggestions.
User-agent: Applebot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# OpenAI: ChatGPT search index. Links back to the source.
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# OpenAI: Live fetch when a user asks ChatGPT about a specific URL.
User-agent: ChatGPT-User
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# Perplexity: Perplexity answer index. Cites sources inline.
User-agent: PerplexityBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# Anthropic: Claude retrieval and citation.
User-agent: ClaudeBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# Anthropic: Live fetch when a Claude user asks about a specific URL.
User-agent: Claude-User
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /account/
# Apple: Apple foundation model training. Opting out does not affect Siri. No referral traffic.
User-agent: Applebot-Extended
Disallow: /
# OpenAI: Model training corpus. Separate from OAI-SearchBot. No referral traffic.
User-agent: GPTBot
Disallow: /
# Google: Gemini training. Blocking it does not remove you from Search. No referral traffic.
User-agent: Google-Extended
Disallow: /
# Anthropic: Legacy training crawler identifier. No referral traffic.
User-agent: anthropic-ai
Disallow: /
# Common Crawl: Open crawl corpus used as training data by many labs. No referral traffic.
User-agent: CCBot
Disallow: /
# ByteDance: Training corpus. Widely reported to ignore robots.txt. No referral traffic.
User-agent: Bytespider
Disallow: /
# Ahrefs: Backlink index. Allowing it makes your own backlink data visible to you and to competitors. No referral traffic.
User-agent: AhrefsBot
Disallow: /
# Semrush: Competitive index. Same trade as AhrefsBot. No referral traffic.
User-agent: SemrushBot
Disallow: /
# Internet Archive: Wayback Machine preservation. No referral traffic.
User-agent: ia_archiver
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
Host: www.example.com
How to use it
- 01Read what each crawler returnsSearch crawlers send clicks. Citation crawlers name the source. Training crawlers return neither.
- 02Choose per crawlerThe default allows search and citation crawlers and denies training-only ones, which keeps answer engine visibility while removing the cost with no return.
- 03List your private pathsThese are repeated under every named agent, because a named agent inherits nothing from the wildcard block.
- 04Copy the outputEither the text file or the generated route, so the file cannot drift from your route constants.
Questions
Should I block AI crawlers?
Block training-only crawlers if you want to; allow citation crawlers in every case. Citation crawlers such as OAI-SearchBot, PerplexityBot and ClaudeBot link back to the source, so blocking them is the answer engine equivalent of blocking Googlebot.
Does blocking GPTBot remove me from ChatGPT?
Blocking GPTBot does not remove a site from ChatGPT's search results. GPTBot fetches training data and OAI-SearchBot fetches the search index. They are separate user agents with separate rules.
Why does the output repeat the disallow list?
A named user agent block in robots.txt inherits nothing from the wildcard block. If a named agent has no disallow list, it is permitted everywhere.
Where this comes from
This tool uses the same modules SEO for Solos sells in the Code Kit, so the output and the product cannot diverge. The reasoning behind it is in the playbook.