The AI crawler decision framework — every robots.txt line priced in four currencies

The one-line version: AI crawlers come in three lanes — training, retrieval, and user-fetch — and each vendor has separate user-agents for each lane. Blocking the wrong one costs you citations without any of the training benefits people think they’re getting. MaxAEO’s audit of 412 B2B SaaS domains over 90 days found 37.9% blocking at least one AI user-agent, and 60.1% of GPTBot-blockers accidentally blocking retrieval bots too. The measurable cost: a 6x drop in ChatGPT citation share (from 12.8% to 2.1%) for domains that blocked OAI-SearchBot. Here’s how to price every rule correctly.

The three lanes

Every AI vendor operates crawlers in three distinct roles, per MaxAEO’s framework:

Cloudflare’s network analysis of the first week of August 2025 quantified the traffic mix:

• Training accounted for roughly 80% of AI bot crawling
• User-action fetches accounted for under 5%
• Anthropic crawlers made ~50,000 requests per referral sent back
• OpenAI’s ratio was 887:1
• Perplexity’s ratio was 118:1

The four currencies to price every rule

Every robots.txt line has costs and benefits in four currencies:

1. Training exposure — whether your content can be fetched for future model pretraining
2. Citation eligibility — whether your site can enter an AI answer engine’s retrieval index and citation shortlist
3. Live-answer accuracy — whether an assistant can fetch your current page during a user session
4. Referral traffic — whether AI answers can send visitors through links back to your site

Most published guides only price training exposure, which is why so much advice reads as confidently wrong. Blocking GPTBot doesn’t affect your ChatGPT citations (that’s OAI-SearchBot’s job) — but many “AI crawler blocking” guides conflate the two.

The vendor user-agent map

The three-lane structure across all major vendors, per MaxAEO’s audit:

Tokens that block nothing (despite being commonly used): ChatGPT, OpenAI, GPT-4, Claude, AI, GPTBot/1.2. These aren’t published vendor user-agents — the rule matches nothing and gives you a false sense of security. Verify every token against vendor documentation.

The rule cost ledger

Every rule you might add, priced across all four currencies:

The audit that quantified the cost

MaxAEO’s audit methodology:

• 412 tracked B2B SaaS and technology domains
• Robots.txt files fetched April-June 2026
• 90 days of daily answer tracking
• ~187,000 recorded answers
• 2,100 buyer-intent prompts
• Tested across ChatGPT, Perplexity, Google AI Mode, Claude, and Copilot

Key findings:

• 37.9% of domains disallowed at least one AI user-agent
• 60.1% of GPTBot-blockers also disallowed at least one retrieval agent (usually accidentally)
• 11.4% targeted unpublished/fictional tokens that block nothing
• 8.3% accidentally opened protected paths through named-group behavior (median 6 exposed paths per affected domain)

The matched-pair citation results (41 pairs, one blocked GPTBot only, one fully open):

• GPTBot-blocking group: 12.4% ChatGPT citation rate
• Fully open group: 13.1% ChatGPT citation rate
• Difference: 0.7 percentage points — statistically indistinguishable

Blocking GPTBot cost basically nothing. Now the OAI-SearchBot comparison:

• Domains blocking OAI-SearchBot: 2.1% ChatGPT citation rate
• Allowed peers: 12.8% citation rate
• The 6x gap in citations that actually link back to the site

The residual 2.1% citation rate for OAI-SearchBot blockers came from parametric memory and third-party pages (review sites, directories, Reddit threads, comparison posts). Unlinked brand mentions held at 9.6% — ChatGPT still discussed those companies, but described them in someone else’s words and sent zero traffic.

The recommended base configuration

If you want to stay out of training corpora while remaining eligible for citations across every major AI engine, from MaxAEO’s exact template:

# Training crawlers — no citation surface attached
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

# Retrieval crawlers — these feed AI citations. Keep open.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
Allow: /
Disallow: /wp-admin/
Disallow: /internal-search
Disallow: /*?sort=

# User-triggered fetchers — one page, one live human question
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: meta-externalfetcher
Allow: /
Disallow: /wp-admin/

# Google-Extended is training + Gemini grounding consent.
# It does NOT affect AI Overviews or AI Mode. Enable only if you
# accept reduced Gemini grounding use.
# User-agent: Google-Extended
# Disallow: /

User-agent: *
Disallow: /wp-admin/
Disallow: /internal-search
Disallow: /*?sort=

Sitemap: https://example.com/sitemap.xml

Three things this configuration does that matter:

1. Multiple User-agent lines per group is valid syntax — it keeps the three lanes visually separate and easy to audit
2. Protective paths (/wp-admin/, /internal-search, /*?sort=) are repeated in every named group because named groups don’t inherit from User-agent: *
3. Sitemap declaration at the bottom — universally supported crawl-priority signal

The failure pattern most sites have

The most common configuration bug in MaxAEO’s audit — the wildcard-group inheritance mistake:

User-agent: *
Disallow: /wp-admin/
Disallow: /internal-search
Disallow: /*?sort=

User-agent: OAI-SearchBot
Allow: /

Intended meaning: allow OAI-SearchBot generally while retaining wildcard protections.

Actual effect: OAI-SearchBot is now allowed to crawl /wp-admin/, every internal search page, and every sort-parameter permutation. Once a crawler matches a specific User-agent line, it stops reading the * group entirely. Named groups don’t inherit.

The fix — repeat the protective rules inside every named group:

User-agent: OAI-SearchBot
Allow: /
Disallow: /wp-admin/
Disallow: /internal-search
Disallow: /*?sort=

The Google exception

Google-Extended is the most misunderstood token in the entire AI crawler landscape. Per MaxAEO:

• It does NOT block AI Overviews — those use ordinary Googlebot
• It does NOT block AI Mode — same story
• It DOES block Gemini/Vertex training exposure and some grounding use

There is no robots.txt token that removes you from AI Overviews while keeping you in the ten blue links. Blocking Googlebot is the only way, and it’s catastrophic — you lose Search AND AI Overviews AND AI Mode simultaneously. This is why Google-Extended is commented out in the base config: enable it only if you deliberately want to reduce Gemini grounding exposure and accept that trade.

Compliance quirks to know

Perplexity-User doesn’t honor disallow rules. Perplexity states that Perplexity-User acts on a user’s behalf rather than as a crawler, so it treats robots.txt disallows as non-binding. Blocking it in robots.txt doesn’t actually stop it. See Perplexity source selection for the deeper mechanics.

Anthropic states all three Claude agents honor robots.txt — including Claude-User (unlike Perplexity’s equivalent).

Case sensitivity split: User-agent matching is case-insensitive (gptbot and GPTBot work the same). Path values are case-sensitive.

Crawl-delay is not honored by Googlebot or most AI crawlers. Rate control needs to happen at your edge (Cloudflare, Fastly, nginx) rather than through robots.txt directives.

The recovery data — unblocking works

Across nine domains in MaxAEO’s audit that removed a retrieval-agent disallow during the tracking window:

• Median time to first new ChatGPT citation: 11 days
• Range: 3 to 34 days

The fast cases:

• Resubmitted a sitemap with updated lastmod dates immediately after unblocking
• Linked newly opened sections from a page that was already crawled frequently

The slow cases:

• Thin internal linking
• No fresh sitemap signal

If you find you’ve been blocking retrieval bots, the fix is fast: update robots.txt, resubmit the sitemap in Google Search Console and Bing Webmaster Tools (see the verification guide), and add IndexNow submission on rebuild. Median 11 days to first new citation.

The verification workflow

After deploying any robots.txt change:

1. Fetch and archive the current file with a timestamp
2. Classify every directive: training / retrieval / user-fetch / traditional search / unknown scraper
3. Delete tokens no vendor publishes (ChatGPT, OpenAI, GPT-4, etc.)
4. Copy wildcard-group protective paths into every named group
5. Deploy during low-traffic window
6. Grep server logs for expected user-agents (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot)
7. Verify source IPs against each vendor’s published IP range files (OpenAI, Anthropic, Google, Perplexity all publish them)
8. Confirm HTTP responses are 200, not 403/429/challenge pages
9. Measure answer visibility for 30 days before trusting the result

A common failure mode: a Cloudflare challenge interstitial returns a successful HTTP status while containing none of your content, so the crawler logs a hit but indexes nothing. This can look like normal traffic but produce zero citations.

The strategic point

Every robots.txt rule you add is a decision priced in four currencies. Most sites are running configurations that block the wrong bots (blocking retrieval crawlers that drive citations) while allowing the wrong ones (leaving training bots that produce zero visibility open, or targeting fake tokens that block nothing).

The right base config takes ~30 lines, keeps your citation surface intact across every major AI engine, blocks training exposure where you actually want to, and repairs the wildcard-group inheritance trap that 8.3% of audited sites had. Deploy it, verify the logs, and re-measure citations in 30 days. If you were accidentally blocking retrieval, the median 11-day recovery window means the payoff comes fast.

Related reading: ChatGPT Stage 1 in depth — Bing indexation, OpenAI’s three crawlers, IndexNow. Perplexity source selection — how PerplexityBot’s crawling drives the reranker’s shortlist. Search Console verification — the sitemap resubmission step for the 11-day recovery playbook.