ChatGPT Stage 1 in depth: Bing indexation, OpenAI's three crawlers, and IndexNow

The one-line version: ChatGPT Search retrieves exclusively from Bing’s index (Machine Relations), OpenAI operates three separate crawlers with three different purposes, and IndexNow is the single fastest way to get URLs into Bing’s index — under 10 minutes end-to-end with the right key file. If Stage 1 of the ChatGPT three-stage pipeline fails, nothing else you do matters. Here’s how to make sure it doesn’t.

Why Stage 1 gets skipped

Most GEO content jumps straight to passage-selection tactics — direct-answer paragraphs, H2/H3 sub-questions, schema — and treats indexation as a solved problem. It isn’t. ChatGPT can only retrieve pages Bing has indexed, and Bing indexes far less of the web than Google does. Priority URLs that rank on page one of Google are often missing entirely from Bing.

The technical work is small, mostly one-time, and comes with a compounding advantage: every URL you add to Bing becomes eligible for ChatGPT Search citation, and the marginal cost per URL after the initial setup is near zero.

OpenAI’s three crawlers

Per OpenAI’s own documentation, OpenAI operates three distinct crawlers with three purposes. Each one can be independently allowed or blocked in robots.txt:

Published IP address lists live at openai.com/searchbot.json, openai.com/chatgpt-user.json, and openai.com/gptbot.json.

Two important nuances:

1. These settings are independent. You can allow OAI-SearchBot to surface in ChatGPT Search while disallowing GPTBot to keep your content out of training data.
2. Bot allow-listing takes ~24 hours to propagate. OpenAI states that changes to robots.txt take approximately 24 hours before OpenAI’s systems adjust.

For GEO purposes, OAI-SearchBot is the only crawler that determines your visible-citation eligibility in ChatGPT Search. ChatGPT-User is a user-triggered live fetch that generally ignores standard crawl rules (users have already asked ChatGPT to fetch this page). GPTBot is a long-term training-data question, covered in the brand in training data article.

The minimal robots.txt for ChatGPT Search

Add these blocks near the top of your robots.txt file, above any User-agent: * fallback (so more specific rules take precedence):

# OpenAI — allow search retrieval, allow user-triggered fetches
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# GPTBot handles training data — set to your policy
User-agent: GPTBot
Allow: /

If you want to opt out of training-data crawling while remaining eligible for ChatGPT Search citations, change the last block to Disallow: /. The other two stay.

Also add the two Perplexity user agents while you’re here — see the Perplexity pillar for the reasoning:

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Perplexity’s official crawler docs are at docs.perplexity.ai/docs/resources/perplexity-crawlers.

Bing indexation — the real bottleneck

Bing indexation lags Google materially for most sites. The three ways to close that gap, in order of effectiveness:

1. IndexNow (highest leverage — you should do this)

IndexNow is an open protocol supported by Bing, Yandex, Naver, Seznam, Yep, Amazon and others. Submitting a URL to any participating endpoint shares the submission with all IndexNow-enabled search engines. Bing’s endpoint is:

https://www.bing.com/indexnow?url={your-url}&key={your-key}

The generic API-wide endpoint is https://api.indexnow.org/indexnow?url={your-url}&key={your-key}.

Setup, one time:

1. Generate an 8-128 character API key using only a-z, A-Z, 0-9, and -. Example: 4f9c2a7b3d8e6f1a.
2. Save a UTF-8 text file at your web root: https://yourdomain.com/{your-key}.txt. The file must contain only the key itself and must be publicly accessible with no login.
3. Verify the key file loads correctly: curl https://yourdomain.com/{your-key}.txt should return only the key.

Ongoing use:

• Submit each new or updated URL to Bing’s endpoint. Every 200 response means the URL was received; 202 means received but not yet processed. Subsequent submissions typically return 200 after first-request verification.
• For a bulk push (up to 10,000 URLs per request), POST to https://api.indexnow.org/indexnow with Content-Type: application/json; charset=utf-8 and a body of:

{
  "host": "www.example.com",
  "key": "4f9c2a7b3d8e6f1a",
  "urlList": [
    "https://www.example.com/url1",
    "https://www.example.com/folder/url2"
  ]
}

• Wait at least 5 minutes between resubmissions of the same URL unless content has materially changed. HTTP 429 = rate limited; respect Retry-After.
• Every subdomain is treated as a separate host, needing its own key file at its own root (IndexNow.org FAQ).

For a WordPress site, install the official IndexNow plugin — it automates submission on publish and update. For a static site (like an Eleventy build), the build pipeline should POST the sitemap-delta to api.indexnow.org on every deploy. A ~20-line Python script covers the workflow.

2. Bing Webmaster Tools sitemap submission

Log in to Bing Webmaster Tools, verify your domain (via a DNS TXT record or a metatag), and submit your sitemap.xml. This is the traditional path and it still works — Bing generally acts on submitted sitemaps within days to a couple of weeks.

Two auxiliary features worth using:

• URL Submission (up to 10,000 URLs per day per site) — a faster alternative to sitemap re-crawling.
• URL Inspection — shows whether a specific URL is indexed and, if not, why. This is the diagnostic tool for the “why isn’t ChatGPT citing this page?” question.

3. External signal building (slower path)

Bing weighs domain-external signals more heavily than Google does in some categories. If a page is failing to index despite IndexNow submission and Bing Webmaster verification, adding 1-3 external referring domains to that specific URL usually unblocks it within a couple of crawl cycles.

The full Stage 1 checklist

Before optimizing any content for ChatGPT citations, verify every priority URL passes this list:

• [ ] robots.txt allows OAI-SearchBot, ChatGPT-User, GPTBot (unless training-opt-out is your policy — then GPTBot: Disallow: /).
• [ ] robots.txt allows PerplexityBot and Perplexity-User for the sibling engine.
• [ ] Bing Webmaster Tools shows the domain verified.
• [ ] sitemap.xml submitted in Bing Webmaster Tools.
• [ ] IndexNow key file live at https://yourdomain.com/{your-key}.txt.
• [ ] IndexNow submission wired into your publish/deploy pipeline (WordPress plugin or ~20-line script for static sites).
• [ ] Bing URL Inspection shows every priority URL as indexed.
• [ ] Server logs show OAI-SearchBot hits within 24-72 hours of a new page publishing.
• [ ] Client-side JS is not required to see the primary content — server-rendered HTML.
• [ ] No 4xx / 5xx / broken redirect chains on priority URLs.

Every item above is a Stage 1 gate. Miss any of them and no amount of Stage 2 (passage selection) or Stage 3 (citation decision) work will save you.

The compounding advantage

The setup above is one afternoon of work. After that, every new URL you publish is automatically pushed to Bing (and via IndexNow to Yandex, Naver, and every other participating engine) within minutes of publishing. Compared to a Google-only publishing pipeline, Bing indexation happens faster, and the marginal cost per URL falls to essentially zero.

That’s the arbitrage. Most sites still don’t use IndexNow, don’t verify in Bing Webmaster Tools, and don’t allow-list OAI-SearchBot. Getting these right puts you in a smaller pool of eligible pages for ChatGPT to cite.

The ChatGPT pillar page puts this Stage 1 work into the wider three-stage pipeline. The ChatGPT citation signals article covers Stages 2 and 3.

Related reading: What actually gets ChatGPT to cite you — the full three-stage pipeline. Getting your brand into LLM training data — the 60% of ChatGPT answers that don’t use live retrieval and can’t be fixed with indexation.