What is llms.txt

The name imitates robots.txt, and that resemblance creates the first confusion. robots.txt governs permissions: it tells a crawler which paths it may request. The XML sitemap solves discovery: it lists the URLs the owner wants to be found. llms.txt does neither. It is an orientation index, written in Markdown, meant for a language model that already has the site in front of it and needs to see at a glance which documents matter and what each one is for.

The proposal comes from Jeremy Howard and was published on 3 September 2024. Since then it has circulated as a living specification, now in a second version of the text. It has not passed through any standards body, there is no official register for the file, and no search engine has added it to its crawling documentation. Calling it a standard is a category error: it is a convention one author proposed and part of the community adopted.

It should also be kept apart from llms-full.txt, a dump of the entire content into a single file. That variant circulates by community habit and does not appear in the specification, so it does not even share the backing of a published document.

The file lives at the root of the domain, at the path /llms.txt, and follows a fixed structure. It starts with an H1 heading carrying the project name, the only mandatory element. Below it sits a blockquote with a short summary. Free prose without headings may follow. The rest consists of H2 sections, each with a list of Markdown links; after each link a colon and an explanatory note are allowed. A section called Optional conventionally holds the secondary material, the part a model can skip when context runs short.

# Acme Docs

> Documentación de producto de Acme.

## Guías
- [Inicio](https://acme.com/inicio)
- [API](https://acme.com/api): referencia

## Optional
- [Changelog](https://acme.com/changelog)

That is as far as the documented mechanism goes. The part almost nobody tells is the consumption side. Google spoke on 23 July 2025, through Gary Illyes at a Search Central Deep Dive event: Google does not support llms.txt and does not plan to.

Server logs point the same way. An analysis of 137,210 domains covering May 2026 found that 97% of published llms.txt files received not a single request that month. Of the requests that did arrive, most came from SEO audit tools and technology profilers rather than AI systems. And on domains without the file, no AI bot went looking for it: they do not probe for its existence. The same analysis found that 28% of those 137,210 domains publish an llms.txt file, all of them running the same analytics tool, which makes the population more technical than the web at large. Adoption exists. Reading is what is missing.

The decision hanging on this term is easy to state and easy to answer badly: is the file worth publishing? The honest answer is neither yes nor no. Creating it costs little, roughly an hour for a medium-sized site, and as of today there is no public evidence that it improves visibility in generated answers. Anyone publishing it should do so knowing that, not expecting traffic.

The real risk is not the file but what it displaces. A team that spends its AI visibility budget maintaining a Markdown index while leaving the HTML behind unrendered JavaScript, or while accidentally blocking a provider's search crawler, is optimising the part nobody reads and neglecting the part everybody crawls. AI systems that cite pages reach them the normal way: crawling, index, retrieval.

There is a second cost, maintenance. An outdated llms.txt points at URLs that no longer exist and describes sections that were renamed. If providers ever do start reading it, the abandoned file becomes a source of errors rather than an advantage. Publishing it means adding it to the site's review routine, alongside the sitemap and the redirects.

• Publish the file only if you will maintain it: add it to the same periodic review as the sitemap, the redirects and the canonicals.
• Serve it as plain text in UTF-8 and check that it answers 200 at the domain root, not in a subdirectory and not behind a redirect.
• Link only canonical URLs reachable without a logged-in session; a list of broken links is worse than having no file at all.
• Write each link's note for someone who does not know the site: what the document contains and which case it serves.
• Measure before you opine: filter your access logs by the path /llms.txt and see which agents request it on your own domain.
• Treat it as a later extra, once the HTML renders without JavaScript and robots.txt lets the AI providers' search crawlers in.

• Presenting it as a standard in a proposal or a client audit, when it is a proposal with no uptake by the search engines.
• Replacing the XML sitemap with llms.txt. They are different mechanisms, and the sitemap really is consumed by search engines.
• Generating it with a plugin and never looking at it again, so it ages with every structural change to the site.
• Crediting it with rises in ChatGPT or Perplexity mentions that coincide in time with other changes; without a control group no cause is proven.
• Dumping the whole site into llms-full.txt in the belief that this trains the model, when training depends on crawling and permissions.