Do AI Crawlers Respect robots.txt? What PR and SEO Teams Need to Know

Do AI crawlers respect robots.txt? It is the question every PR and SEO lead asks once they realise that ChatGPT, Claude and Perplexity are now front-door discovery channels, not a future concern. The short answer is reassuring: the reputable AI vendors document and honour robots.txt for their named crawlers. The longer answer is where the mistakes happen, because 'AI crawlers' is not one thing — it is a set of bots with different jobs, and a rule written for one can quietly disable another.

Do AI crawlers respect robots.txt? The short answer

Yes — the reputable vendors document and honour it for the crawlers they name. The rest of this guide explains which crawlers those are, why a careless rule can still take you out of AI answers, and what a safe policy looks like in practice. The reassurance has a condition: you have to write the rule for the bot you actually mean, not for 'AI' as a single block.

Which AI crawlers respect robots.txt

The vendors people actually mean when they say 'AI' publish named user-agents and explain their behaviour. OpenAI runs GPTBot for training and OAI-SearchBot for ChatGPT's search answers; both are documented as respecting robots.txt. Anthropic runs ClaudeBot (training) and Claude-SearchBot (retrieval), also documented as honouring it. Perplexity's PerplexityBot, which indexes pages for its answers, respects robots.txt; Google's Google-Extended token, which governs Gemini training, is honoured and is separate from Google Search; Apple's Applebot-Extended and Common Crawl's CCBot follow the same pattern. So to the direct question — do AI crawlers respect robots.txt — the answer for every major lab is yes, for the crawlers they name.

Training crawlers versus retrieval crawlers

The distinction that matters most is the job. A training crawler sweeps the web to build or improve a model; blocking it opts you out of future training, with no effect on how any product answers about you today. A retrieval crawler indexes pages so an AI product can cite or link to them in real time — it is the AI equivalent of a search engine. These are different bots under different user-agent tokens, and a policy that treats them as one blob usually breaks the wrong one. Our deeper look at GPTBot versus OAI-SearchBot covers the OpenAI split in detail, and the principle holds across vendors: each bot is its own switch.

The blanket Disallow: / trap

This is the expensive mistake. Someone writes 'User-agent: *' with 'Disallow: /' to keep AI away, or copy-pastes a block list that includes the retrieval bots, and the site vanishes from AI answers entirely — not just from model training. Because retrieval and answer bots are what make your content citable, blocking them removes you from ChatGPT Search, Claude and Perplexity citations in one stroke. The right shape is selective: disallow the training crawlers you do not want feeding models, and leave the retrieval crawlers allowed so your pages can still be cited. For background on why citations depend on crawl access rather than on new files, see our notes on what llms.txt is and is not.

User-triggered fetches are a different case

A second nuance: when a person explicitly asks an assistant to open a specific URL, the fetch is treated like a user clicking a link. OpenAI's ChatGPT-User and Perplexity's Perplexity-User are documented as generally ignoring robots.txt for those on-demand fetches, because a human asked for the page. That is a deliberate exception, not a bug, and it means robots.txt cannot fully suppress a page a user specifically requests. For most publishers this is acceptable — you would not want an assistant to refuse a link someone pasted in on purpose — but it is worth knowing the control is not absolute even for the labs that honour the file.

What robots.txt can and cannot do

robots.txt controls fetching. It tells a crawler which paths it may request; it does not, and never could, dictate what a crawler may do with the bytes afterwards. There is no machine-readable way to say 'you may read this to answer a question but not to train on it' — that line is exactly what the industry is still trying to standardise. robots.txt is also a request, not a firewall. The major labs state they follow it and the evidence says they largely do, but unidentified scrapers ignore it completely. For those, the defence is at the server or CDN layer — rate limits, WAF rules, user-agent blocks — not a text file anyone can ignore.

A safe default policy

For a publisher or marketing site that wants to stay citable but limit training use, a reasonable starting point is to disallow the training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider) and allow the retrieval and live bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the user-fetch agents). Google Search and Bing are unaffected because they run on Googlebot and Bingbot, not the AI training tokens. Re-check the vendor docs periodically, because user-agent names change; a stale list from 2023 will miss the retrieval bots that matter most for visibility today. The broader mechanics of getting cited by AI assistants still rest on crawl access first.

Frequently asked questions

Two questions close the loop. First, does this affect paid visibility? No — robots.txt governs crawling, not advertising, and blocking a training crawler has no measured effect on your classic search rankings. Second, should a small site bother? If your content is the kind of thing people ask assistants about, yes: a selective policy keeps you in AI answers while limiting how your text feeds models, and it costs nothing but a few correct lines. The principle behind how AI-ready distribution is priced is the same — visibility is a setting you choose, not a default you hope for.

So, do AI crawlers respect robots.txt? The credible ones do, for the crawlers they name. The risk is not that they ignore you; it is that a clumsy rule ignores them on your behalf — and quietly takes you out of the answers your audience is already asking for.