What Is llms.txt? The Honest Guide for PR and Content Teams
What is llms.txt — the Markdown file proposed as a guide for AI agents — and does anything actually read it? A plain-English look at the format, the documented crawler behaviour of OpenAI, Anthropic and Google, and the levers that really drive AI citations.
Key takeaways
- llms.txt is a community proposal (September 2024, llmstxt.org) for a small Markdown file at your site root that gives AI agents a curated map of your content; a second draft revision landed in August 2026.
- As of late 2026 no major AI platform documents reading it: OpenAI, Anthropic and Perplexity crawler docs name only robots.txt, and Google has publicly said it does not use the file, comparing it to the long-ignored keywords meta tag.
- The AI labs publishing llms.txt for their own documentation proves producing, not consuming; server logs show occasional fetches, and studies of large samples of AI citations have found essentially none pointing to llms.txt files.
- Citations are driven by documented levers: robots.txt crawl access (mind the split between training and retrieval crawlers), extractable content structure, FAQPage schema, and authority earned from trusted sources.
- Ship an llms.txt for docs-heavy developer audiences if you like; for a marketing or PR site treat it as an unproven extra, not a substitute for the work that demonstrably earns citations.
Ask 'what is llms.txt' and you will meet two confident, opposite answers. A growing plug-in ecosystem treats the file as a must-have for AI visibility; the search engines themselves say it does nothing for them. Both camps are partly describing reality, which is why this guide sticks to what can be verified: what the file actually contains, which AI systems have documented reading it, and what still decides whether an AI assistant cites your page.
What is llms.txt?
llms.txt is a community proposal, first published by Jeremy Howard of Answer.AI in September 2024, for a single Markdown file served at the root of a website. The format is deliberately small: an H1 title, a one-paragraph summary in a blockquote, then Markdown link sections grouped by topic, each link carrying a short description. A sibling file, llms-full.txt, can concatenate the full text of the linked pages for agents that want everything in one pass. The specification lives at llmstxt.org and received a second draft revision in August 2026.
The intended reader is an AI agent with a limited context window that has just landed on your site. Instead of crawling hundreds of pages to work out what matters, it reads one small file that says: here is what this site is, here are the pages that matter, and here is what each contains. That is the whole idea. Nothing in the proposal promises citations, rankings or traffic — only that agents can orient themselves faster.
How llms.txt differs from robots.txt and sitemap.xml
Because all three files live at the root of a domain, they get lumped together. They do different jobs and sit at very different levels of maturity.
| File | Job | Status |
|---|---|---|
| robots.txt | Controls which crawlers may fetch which paths | An internet standard (RFC 9309), respected by essentially every crawler |
| sitemap.xml | Tells crawlers which URLs exist to index | A widely supported protocol used by all major search engines |
| llms.txt | Offers agents a curated overview of the site | A community proposal; no standards body and, so far, no major engine documenting that it reads one |
The maturity column is the part that matters for planning. robots.txt is a lever that provably works; llms.txt is a bet that has not paid out yet, at least not in any way the publishers of AI systems have acknowledged.
Does any AI engine actually read llms.txt?
As of late 2026, the documented answer is no. OpenAI's crawler documentation for GPTBot, OAI-SearchBot and ChatGPT-User names robots.txt as the control surface and does not list llms.txt as an input. Anthropic's crawler policy likewise describes robots.txt directives and says nothing about its bots consulting llms.txt. Perplexity's bot documentation is the same. Google has been the most explicit of all: its documentation for AI features states that you do not need to create new machine-readable files or AI text files to appear in them, and Google's search representatives said publicly in 2025 that the file is comparable to the long-ignored keywords meta tag and that Google neither uses it nor plans to.
Two observations keep the debate alive, and both are worth reading carefully. First, the AI companies themselves publish llms.txt files for their own developer documentation — Anthropic, OpenAI, Perplexity and Cloudflare all serve one. That is producing, not consuming: a courtesy to the coding agents that read API docs all day. It is not evidence that their crawlers treat your llms.txt as an input. Second, server-log analyses have shown major AI crawlers occasionally requesting /llms.txt. Fetching a file is not the same as using it for discovery or citation, and industry studies of large samples of AI citations have found essentially none pointing at llms.txt files. The gap between 'the file gets requested' and 'the file drives visibility' is exactly where most of the sales copy lives.
Why the file spread anyway
The economics explain the adoption curve. Shipping an llms.txt costs an afternoon and carries no downside — no engine penalises you for having one, and a well-written file occasionally helps a human developer too. Documentation platforms began generating one automatically (Mintlify in late 2024), WordPress SEO plug-ins added generators, Lighthouse started auditing for it, and answer-engine-optimisation checklists began listing it as a box to tick. By 2026, hundreds of thousands of sites serve an llms.txt, with the heaviest adoption in developer documentation and almost none of it driven by evidence of citation lift.
What actually decides whether AI cites your page
If llms.txt is not the lever, what is? The same things that decide whether any crawler can use your content. Crawl access comes first: the documented control surface is robots.txt, and the crawlers are not interchangeable. Training crawlers and retrieval crawlers are separate agents with separate names, so a blanket block that feels like protecting your content can quietly remove you from the very answers you want to appear in. We cover that split in GPTBot vs OAI-SearchBot.
Above crawl access sits content quality in the extractable sense: clear headings, answers stated plainly near the top of sections, verifiable numbers, and structured data such as FAQPage that machines can parse without guessing. And above all of that sits the factor no file can fake — being quoted and linked by sources the models already trust. The full picture is in how AI assistants choose what to cite. None of these levers is new; llms.txt is best understood as a small, unproven addition to a stack that already works.
Should your site ship an llms.txt?
A reasonable default for late 2026: if your site is documentation-heavy and read by developers who wire content into agents and IDEs, ship one — it is cheap, some agentic tools use such files opportunistically, and it doubles as a tidy overview for humans. If your site is a marketing or PR destination and the file would be generated by a plug-in and never maintained, the expected lift is close to zero, and the hour is better spent making one high-value page genuinely citable. If you do ship one, keep it under roughly 10 KB, write a real description for every link instead of dumping bare URLs, update it when the site changes, and never treat it as a substitute for crawl access, structured data or actual authority.
The next step after understanding what is llms.txt is boring and effective: pick the page you most want quoted, check that the right retrieval crawlers can reach it, state its key answer in the first paragraph, and link it from pages that already rank. The rest of our insights on China PR, search and AI visibility covers the specifics.
Frequently asked questions
Does Google use llms.txt?
No. Google's documentation states that its AI features do not require new machine-readable files or AI text files, and Google search representatives said publicly in 2025 that Google does not use llms.txt, comparing it to the deprecated keywords meta tag.
Is llms.txt an official web standard?
No. It is a community proposal published at llmstxt.org in September 2024 and revised in August 2026. No standards body has adopted it and no major AI engine has documented that its crawlers read the file.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a small curated index: a title, a one-paragraph summary and described links grouped by topic. llms-full.txt concatenates the full text of those pages into one file for agents that want complete context in a single pass. Most generators produce both.
Will adding llms.txt improve my ChatGPT citations?
There is no reliable evidence that it does. Analyses of large samples of AI citations have found essentially none pointing to llms.txt files. The documented levers are crawl access via robots.txt, well-structured extractable content and authority.
Do AI crawlers check llms.txt instead of robots.txt?
No. robots.txt remains the documented and honoured control surface for every major AI crawler, both training and retrieval. Server logs show some crawlers occasionally fetch /llms.txt, but fetching a file is not the same as using it for discovery or citation.