What is llms.txt?
Jeremy Howard published the proposal on 3 September 2024 (Answer.AI). The rationale, according to llmstxt.org: despite their growth, language model context windows are still too small for most websites in their entirety, and every wasted token costs time and money. A curated overview in Markdown is intended to help agents find the relevant content quickly without processing complete HTML pages with navigation and scripts.
The file is mainly intended for the moment an AI system answers a question or carries out a task, rather than for training models. A second version of the proposal appeared on 10 August 2026 (llmstxt.org), in which Howard writes that thousands of websites now publish an llms.txt file.
How an llms.txt file is structured
The file sits at /llms.txt and, according to the specification, optionally in a subdirectory too. It is written in Markdown, not HTML, and contains:
- a first-level Markdown heading (
# Name) with the name of the website or project, the only required element, - a short summary as a Markdown blockquote (
>), - optional further explanations,
- sections with second-level Markdown headings (
## Section) containing lists of links, - by convention, a section called “Optional” for less important content.
A simplified example:
# Example Ltd
> Manufacturer of office furniture based in Bonn. Advice and installation in Germany and Austria.
## Products
- [Desks](https://www.example.com/desks/): overview, options and prices
- [Office chairs](https://www.example.com/office-chairs/): ergonomics and adjustment options
## Company
- [Contact and locations](https://www.example.com/contact/)
## Optional
- [Press](https://www.example.com/press/)
Who uses llms.txt and who does not
Google Search: in its guide to generative AI search, Google states that no new machine-readable files, AI text files or Markdown versions are needed for Google Search, including its AI features, because Google Search does not use them. An llms.txt file neither harms nor helps visibility. Google’s John Mueller had already compared the file to the keywords meta tag in April 2025: a site owner’s claim about the site that can simply be checked against the site itself. In his observation, AI services did not even request the file at the time (Search Engine Journal).
Chrome Lighthouse: the testing tool includes a check in its experimental Agentic Browsing category for whether an llms.txt file exists in the root directory (Chrome for Developers). This concerns AI agents in the browser, not search rankings.
AI providers: we have not found any public commitment from OpenAI, Anthropic or Google to use the file for their chat products. OpenAI, Anthropic and Google do, however, publish llms.txt files for their own developer documentation.
What log files show
Whether AI crawlers request an llms.txt file can be checked in the server log files. In one client project, we analysed ten days of log files from the website of a large German organisation, around 12 million lines. The website had no llms.txt file, yet none of the major AI providers’ crawlers even requested one during that period. At the same time, ChatGPT-User, the agent ChatGPT uses to fetch pages for user requests, retrieved regular pages of the website around 2,900 times a day on average. This is an upper limit, because monitoring tools also trigger such requests.
So AI systems do read websites, but they read the regular HTML pages. Their quality and accessibility are what counts.
llms.txt and robots.txt: the difference
robots.txt is an established standard that website owners use to allow or disallow crawler access. OpenAI and Anthropic document how their crawlers can be controlled there (OpenAI, Anthropic). For ChatGPT-User, however, OpenAI notes that robots.txt rules may not apply to requests initiated by a user. llms.txt has no controlling effect. It offers content, it does not set access rules. If you want to block or allow AI crawlers, you still need to do so in robots.txt.
When an llms.txt file is worthwhile
The effort is small, but a measurable benefit for visibility in AI search has not been demonstrated. The file can make sense for documentation and developer sites whose content users deliberately load into AI tools or agents. For company websites, it is optional. If you create one, keep it up to date, because an outdated overview is worse than none.
For GEO, indexable HTML content, unambiguous facts and access for AI providers’ crawlers are more effective. We check these points in our GEO audit.
Would you like to have us check whether AI systems can reach and correctly understand your content and need support? Then get in touch or read more about our GEO audit! Would you like to find out more or have your team trained on this topic? Then take a look at our GEO seminar.
