llms.txt

llms.txt: what the file is, and what the archive records of it

llms.txt is a proposed convention: a Markdown file at the root of a website that describes the site to language models and points them to the pages that matter. It forbids nothing and reserves nothing. This page explains what the file contains, how it differs from robots.txt, ai.txt and a text-and-data-mining reservation, and why what such a file said on a particular day can become a question later. Guides to writing the file are easy to find; this page is about the record of it.

What llms.txt is

The convention was proposed in September 2024 and is published at llmstxt.org. It is a proposal, not a standard of a standards body, and nothing obliges a site to publish the file or an AI system to read it. According to the proposal, the file is named llms.txt and sits at the root of a site (/llms.txt), or under a subpath for the URLs beneath it. It is written in Markdown and has a fixed order:

• a first-level heading with the name of the site or project (the only required part);
• optionally a short summary in a block quote and further explanatory paragraphs;
• any number of second-level sections, each a list of links with a short description, usually to pages in plain Markdown.

The purpose, in the words of the proposal, is to give language models concise, curated information in one place instead of leaving them to work through a site built for human readers. A minimal example for an invented site:

# Example Shop
> Furniture for small flats, shipped within the EU.

## Products
- [Catalogue](https://shop.example/catalogue.md): all items with dimensions and prices
- [Delivery](https://shop.example/delivery.md): countries, lead times, returns

## Optional
- [Press](https://shop.example/press.md): company history and contact

How it differs from robots.txt, ai.txt and a reservation

The files that address machines do different things, and the difference is the point of this page:

• robots.txt tells crawlers, by the name they announce, which paths they may fetch. It restricts.
• ai.txt is a newer, not standardised convention that states a position towards AI systems. It declares.
• A text-and-data-mining reservation (for example in /.well-known/tdmrep.json) expresses that a rights-holder reserves that use. How the law treats it is set out on the page on text and data mining and the machine-readable reservation.
• llms.txt describes. It offers a language model a guided way into the content, and it contains no mechanism to refuse anything. The proposal does not present it as a reservation, and whether it is of any significance under a particular law is a legal question that this page does not answer.

Why its content on a given day can matter

An llms.txt is a statement a site makes about itself, to machines, at a point in time. It can name pages, describe offers, and change from one week to the next without any trace: a server keeps no history, and when the file is edited, the earlier version is gone. A question about it usually comes later and is a plain question of fact: what did this site offer to AI systems about itself on that day, which pages did it point them to, and when did that change? A screenshot taken today shows today, and a server log is the word of one of the parties. What answers the question is a record made on the day itself by someone who did not know it would be asked.

For the invented example above, such a record would show that on 3 March the file listed catalogue and delivery pages, that a press section appeared on a later date, and that on another date the file was no longer there.

What MachineWitness records of it

The archive fetches /llms.txt at the root of every domain it observes, every day, together with robots.txt, ai.txt and tdmrep.json. It does so on both of its independently operated witnesses. For each fetch it stores what was served: the exact bytes, the complete response headers, a SHA-256 fingerprint of the content, and the time in UTC. When the file does not exist, that is recorded too: an HTTP 404 with no payload, which shows that on that day nothing was served at that address. Each day is sealed into a public root the same night and time-stamped outside this archive. What we store lists every address and limit.

Two limits belong here. Only the root address is fetched; an llms.txt under a subpath is not. And the archive records the file as served: it does not read it for meaning, does not check whether the links work, and does not assess what it says.

If you need the record

• Whether a domain is already observed, and since when: the coverage check, free of charge.
• A domain that is not yet observed and should be recorded every day from now on: daily observation.
• What the file said on named days, as a document with instructions to verify it: an evidence extract.

A record can only begin on the day observation starts; it cannot be made for earlier days.