What llms.txt was supposed to be

The proposal is straightforward and, on the face of it, sensible. Put a markdown file at /llms.txt containing a curated, machine-readable summary of your site — what it is, which pages matter, where the canonical explanation of each topic lives. Language models fetch it, understand your site faster and more accurately, and cite you more reliably.

It borrows its logic from robots.txt and sitemap.xml, both of which solved real coordination problems and became universal. The reasoning by analogy is intuitive. That is most of why the idea spread so quickly.

Eighteen months on, there is enough data to check whether it worked.

What the measurements show

Three findings matter, and none of them are encouraging.

• 10.1% of 300,000 domains studied have an llms.txt file
• 39.6% of those files are empty plugin stubs
• 0.1% share of AI crawler visits that request the file
• 0 major AI crawlers that officially commit to consuming it

Crawlers are not fetching it

In a 90-day window covering 62,100 AI bot visits, 84 requests targeted /llms.txt — around one tenth of one percent. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended overwhelmingly skip the file and crawl HTML directly, which they were already good at.

Google declined, explicitly

This is not a case of a standard awaiting adoption. In July 2025 Google confirmed on the record that it does not support llms.txt and has no plans to, with the comparison drawn to the keywords meta tag — a format that was widely implemented, easily gamed, and ultimately ignored.

Most existing files are empty

Of the sites that do have one, close to 40% are plugin-generated stubs with no curated content. The adoption number overstates real usage considerably.

Why the analogy failed: robots.txt and sitemap.xml succeeded because crawlers needed information they could not otherwise get — permission rules and a URL inventory. llms.txt offers a summary of content the crawler is already reading. It solves a problem the consumer does not have.

Why it is still widely recommended

Given the above, the volume of advice telling you to add one deserves explanation. Three reasons, in descending order of legitimacy.

It is cheap and harmless. Writing one takes an hour. Nothing breaks. Faced with a genuinely uncertain landscape, “do the cheap thing in case it matters later” is not unreasonable advice.

It is a legible deliverable. “We implemented llms.txt” is a satisfying line in a status report in a way that “we rewrote the opening paragraph of nine pages” is not — despite the second one being far more likely to affect whether you get cited.

The agent argument. The most defensible case is forward-looking: llms.txt is a business-to-agent play, the first standard way to publish a machine-readable surface that autonomous agents could route on. That future may arrive. It has not arrived yet, and betting an optimisation programme on it would be a mistake.

Adding llms.txt is a rounding error on your AI visibility. Adding a clear answer to the first paragraph of your twenty highest-impression pages is not.

When it is genuinely worth doing

There are narrow cases where the file earns its hour.

• Documentation and developer products. If people paste your docs into an assistant to work with your API, a curated index of canonical pages has real utility — and this is the use case the format was designed around.
• Large sites with genuinely confusing structure. If you have thousands of URLs and several plausible canonical pages per topic, an explicit statement of which one is authoritative may help any consumer that reads it.
• You are already doing everything else. If your schema is complete, your opening paragraphs answer their questions and your FAQ blocks are published, then yes — spend the hour.

What does not justify it: adding llms.txt while your pages have no structured data, no author attribution and no direct answers. That is optimising the signage on a building with no doors.

If you do add one: Keep it hand-written and short. List your genuinely canonical pages with one line of context each. Do not generate it from a plugin — a stub file is worse than no file, because it looks maintained and is not.

What to do with the hour instead

Ranked by observed effect on citation frequency, roughly highest first:

1. Put a direct answer in the first 40–60 words of each section. Extractability is the strongest single lever, and it is pure editing.
2. Publish FAQ blocks with FAQPage markup on your highest-impression pages. Most sites score zero here, which makes it the largest available gain.
3. Add Organization, Product and Author schema. Entity ambiguity is a common reason a model hedges and cites someone else.
4. Add real author attribution with credentials and dates. Trust signals have become the primary filter for AI inclusion, not a tiebreaker.
5. Fix the pages that are cited about you but are not yours. If a directory outranks you as a source on your own category, that is a content brief with a deadline.
6. Then, if you like, write an llms.txt.

None of this is exciting, and none of it will produce a satisfying line item in a status report. It is, however, what the evidence supports.

Sources

• The State of llms.txt in 2026 — aeo.press
• LLMs.txt Guide: What It Does and Doesn’t Do — Derivatex
• llms.txt: The Complete Guide to the AI Crawler Standard — GrowthGPT