Who requests llms.txt? Three log file analyses compared

Key points:

• In our own server logs and in a client project, no major AI crawler requested llms.txt.
• The tänzer:hosting analysis across all its servers attributes 12 of 3,899 requests (0.3%) to AI training.
• Most requests come from SEO and AI visibility tools and from callers with a normal browser user agent.
• OpenAI and Anthropic, on the other hand, regularly fetch robots.txt and normal pages.

llms.txt is based on a proposal by Jeremy Howard from September 2024. The Markdown file is meant to give language models an overview of a website’s most important content. I explain its structure and origins in our knowledge article on llms.txt. This post deals with a simpler question that log files can answer: who actually requests the file?

For this test I analysed the logs of one of our servers from 15 to 22 October 2026, around 5.1 million lines. The server hosts several websites and the logs do not distinguish between them, so the figures apply to the server as a whole.

The logs contain 146 requests for llms.txt. The full file was served in 68 cases, and the remaining requests were redirects, error responses and two partial responses. None of them carries the name of a major AI crawler, not even a spoofed one.

The table only includes requests whose IP address matches the ranges published by the provider. llms.txt was mainly requested by callers with a normal browser user agent (66 requests), SEOJuice (46), PeecAudit from Peec (7) and WebPageTest (7). Tools such as BuiltWith and Dataprovider added a few more.

A broader data set comes from tänzer:hosting (taenzer.de). We have worked with this host for many years, and all Rheinwunder websites and many of our clients’ projects run there. For this article, managing director Sebastian Tänzer analysed the requests for /llms.txt and /llms-full.txt on all tänzer:hosting web servers, anonymised the results and made them available to us. The following figures are as of 2 November 2026.

It counts 3,899 requests on 158 websites, 3,887 of them since July 2026. Only 4 of these websites actually serve an llms.txt file. 146 do not have one, and the remaining 8 return a server error or block the request.

The analysis breaks down the 12 AI training requests into CCBot (9) and Amazonbot (3). The analysis classes requests with a normal browser user agent as scanners, because people practically never open the file. Redirects and script requests, for example via curl or python, are not counted.

The third picture comes from a client project. We analysed ten days of log files from the website of a large German organisation, around 12 million lines. The website had no llms.txt file, and none of the major AI providers’ crawlers requested one during that period.

At the same time, ChatGPT-User, the agent ChatGPT uses to fetch pages for user requests, retrieved regular pages of the website around 2,900 times a day on average. This is an upper limit, because monitoring tools also trigger such requests. So websites are being read, through their regular HTML pages.

The three analyses show who requests the file. They do not show everything:

• Short periods: Our logs cover eight days and the client project ten. At tänzer:hosting, logs from all servers have only been available since 25 October 2026, before that only from some of them.
• Few websites with the file: At taenzer.de, only 4 of 158 websites serve the file. A crawler looking for it would still appear in the log, even with an error response.
• Unclear browser user agents: Log files cannot reliably show who is behind a normal browser user agent.
• Server view only: Anyone who loads an llms.txt file into an AI tool by hand or points an agent at it may not appear as a known crawler. That is why the file can be useful for developer documentation.
• A snapshot: Provider behaviour can change. I have not found any public commitment from OpenAI, Anthropic or Google to use the file for their chat products.

I would not rely on llms.txt today to gain visibility in AI answers. In its guide to generative AI search, Google states that Google Search does not use such files. For the other providers, our data contains no sign of use.

• Company website: The file is optional. It does no harm if you keep it up to date, but these figures do not justify any significant effort.
• Documentation and developer sites: It can be worthwhile here if users load the content into AI tools themselves.
• Higher priorities: indexable HTML content, unambiguous facts and access for AI crawlers in robots.txt. We check these points in our GEO audit.

All you need is your server’s access logs:

1. Keep the logs: At least one week, as many servers only retain logs for a few days.
2. Filter the requests: for example with grep -E '"(GET|HEAD) /llms(-full)?\.txt' access.log
3. Group by user agent: Count the requests per user agent and per status code. Report redirects separately, otherwise a single request counts twice.
4. Check IP addresses: Names can be spoofed. Of the 10,521 requests using the name ClaudeBot in our logs, 4,365 matched the official IP ranges. OpenAI and other providers publish their ranges.
5. Compare: Look at requests for robots.txt and normal pages alongside them. Only then can you see whether a crawler visited the website and skipped the file.

If you would rather not set this up yourself, we can do it for you in our log file analysis.