GPTBot vs OAI-SearchBot: Which One Controls Whether AI Cites You

GPTBot vs OAI-SearchBot is a comparison that most sites never make on purpose. The rule gets inherited from a template, both agents end up under the same wildcard, and nobody notices until someone asks an assistant about the company and the answer cites a competitor. The two crawlers belong to the same operator and do entirely different work — and only one of them decides whether you can be quoted.

This is a practical note on how the crawlers divide up, what each block actually costs you, and how to write the policy deliberately rather than by accident.

GPTBot vs OAI-SearchBot: two crawlers, two jobs

OpenAI operates several agents that share nothing but an operator. They are controlled separately, and the difference between them is the difference between being trained on and being cited.

The naming is the source of the confusion. Both tokens start with the operator's initialism, both appear in the same section of a robots.txt file, and neither name says "training" or "search" in the plain way you would hope. The consequence of mixing them up is asymmetric: one mistake costs you nothing measurable, the other removes you from AI answers silently.

Why so many sites get this wrong

Three patterns account for most of the damage.

1. The blanket block. A site decides "we don't want AI training on our content", writes one rule that covers every AI-related token it has heard of, and takes itself out of AI search as a side effect. The intent was to opt out of training; the result was also opting out of being cited.
2. The copied template. Robots files get copied between projects. A rule that was correct for a paywalled publisher in 2024 is rarely correct for a marketing site in 2026.
3. The unverified assumption. Many teams believe they blocked AI crawlers and never confirm it, so they also never discover that a CDN, WAF or plugin is serving something different from the file they edited.

The fix is not complicated. It is to treat the two decisions as separate, write one rule for each, and then verify the rendered robots.txt is what you think it is.

What each block actually costs

Blocking GPTBot removes your pages from the training pipeline. It does not remove you from ChatGPT search results or citations, because that path runs through a different agent. If your concern is that your content should not become part of a future model, this is the rule that addresses it — and it is the only one that does.

Blocking OAI-SearchBot removes your pages from the retrieval index used for AI answers. Your content can still be summarised from whatever the model already learned, but your domain will not be fetched, linked or cited as a live source. For a company that wants to be found and quoted, this is the expensive one, and it produces no error message, no notification and no visible symptom on the site itself.

Blocking ChatGPT-User is the least useful of the three. These fetches happen because a human asked for that specific page, and OpenAI's own documentation treats them as user-initiated rather than automated crawling — which means robots.txt rules do not map onto them cleanly, and a block mostly produces a failed fetch for someone who was already interested in you. It is not the control point for anything.

A policy that most content-led sites settle on, and which is defensible: allow the search and user agents, decide the training question on its merits, and document the decision so the next person to touch the file does not flatten it into one wildcard.

Writing the robots.txt blocks correctly

Each agent needs its own user-agent block, and the rule must sit under the right one. The two mistakes here are putting a stray allow rule under the wrong agent, and matching on a version number instead of the token.

# Search retrieval — required to be cited and linked
User-agent: OAI-SearchBot
Allow: /

# User-initiated fetches — someone asked about your page
User-agent: ChatGPT-User
Allow: /

# Training — this is the actual training control
User-agent: GPTBot
Disallow: /

Two operational notes. Match on the agent token, not on a version string, because version numbers change periodically and a pinned rule quietly stops matching. And allow roughly a day for a robots.txt change to be reflected, since agent-side crawling schedules are not immediate.

Verification is worth doing properly rather than trusting the user-agent string, which can be spoofed. OpenAI publishes machine-readable lists of the IP ranges each agent crawls from — separate files for the training, search and user agents — and checking a request against the published ranges is the reliable way to confirm that a claim is genuine. If you keep access logs, also confirm that the search agent is actually visiting the pages you want indexed, rather than only your homepage.

The same split exists outside OpenAI

The pattern repeats across vendors, which is why "we blocked the AI crawlers" is almost never a precise statement.

• Anthropic separates a training crawler from search-index and user-request agents, so the three can be controlled independently.
• Perplexity operates an index crawler and a separate user-agent for fetches triggered by a person, and it cites sources with links, which makes the index agent unusually valuable to allow.
• Google exposes a control token for Gemini training and grounding that is not a crawler at all. Blocking it changes nothing about Googlebot or about your Google Search presence — a distinction that is frequently misread as "blocking AI without hurting search".
• Common Crawl runs its own archive crawler, and a great deal of training data has historically been drawn from archives rather than from direct crawling, so a block aimed at one vendor's crawler does not address archive-based reuse.
• Meta likewise separates a training-oriented crawler from a link-fetching agent.

The practical takeaway: audit the policy agent by agent against each operator's current documentation, and write the intent down. Anything expressed as a category ("AI bots") will drift out of date within a year. The wider set of signals that determine whether an assistant quotes you is covered in getting cited by AI assistants.

Three things that do not do what people think

llms.txt. A proposed convention for declaring preferred content to language models. It is cheap to publish and has no downside, but no major provider has committed to consuming third-party copies of it, Google has stated its search systems do not use it and that no new machine-readable file is needed to appear in its generative features, and independent crawler-log studies report that the overwhelming majority of published files are never fetched at all. Publish one if the maintenance cost is trivial. Do not build a plan that depends on it.

IndexNow. A genuinely useful submission protocol supported by Bing and Yandex among others. Google is not a consumer of it, so a release that says IndexNow improves Google indexing is wrong. Google's own Indexing API is limited to two very specific content types and is not a general-purpose submission channel for articles.

The keywords meta tag. Unused as a ranking or retrieval signal by the major engines, and not consulted by AI crawlers as far as any operator has documented.

What to check on your own site this week

1. Fetch your live robots.txt as a crawler would — not the file on disk — and read which block each AI agent actually matches.
2. Confirm the search-retrieval agents are allowed on the sections you want cited, and only blocked where you genuinely need it.
3. Check that the pages you care about are server-rendered or otherwise readable without running JavaScript, since a fetched page with no content in it cannot be quoted.
4. Decide the training question explicitly, and write the decision and its date into the file as a comment.
5. Put one standing query per quarter into the assistants you care about, and note which sources get cited. That list is the only real measurement of AI visibility.

Frequently asked questions

Is GPTBot the same as OAI-SearchBot? No. GPTBot collects pages for model training. OAI-SearchBot indexes pages so ChatGPT's search features can retrieve and link to them. Blocking GPTBot does not remove you from AI citations, and blocking OAI-SearchBot does not stop your content being used for training.

Will blocking GPTBot hurt my AI visibility? Not directly, because citations run through a different retrieval agent. The indirect effect is that your content will not inform future models. If you want to be quoted, the decision that matters is the retrieval agent, not the training one.

Does robots.txt control AI crawlers? The major operators document that their crawlers respect robots.txt for automated crawling, and OpenAI has said changes are typically reflected within about a day. User-initiated fetches are a different case: those happen because a person asked for a specific page, and robots.txt rules do not apply to them in the same way.

Does llms.txt get my site cited by AI assistants? There is no evidence of that. No major provider has committed to reading third-party llms.txt files, Google has said its systems do not use it, and crawler logs show most published files are never requested. It is cheap hygiene, not a visibility lever.

How do I verify a crawler really is who it claims to be? Check the request against the operator's published IP ranges rather than trusting the user-agent string, which can be spoofed. OpenAI publishes separate range files per agent specifically so this check can be automated.

Does blocking Google-Extended affect my Google rankings? No. Google-Extended is a control token for how your content may be used in Gemini training and grounding, not a crawler. Blocking it has no effect on Googlebot or on how your pages perform in Google Search.

A policy you can defend in a year

Write the retrieval decision and the training decision as two separate lines with two separate reasons, put a date on them, and re-read it once a quarter against the operators' current documentation. That is a fifteen-minute habit that prevents the most expensive silent failure in AI search: being absent from the answers because of a rule someone added for an unrelated reason. If you want the broader structure that makes a page worth quoting in the first place — extractable facts, clear attribution, an unambiguous entity — start with the retrieval-versus-ranking breakdown. Getting the crawler policy right only matters once the page is worth fetching.