Text and data mining

Text and data mining and the machine-readable reservation

EU law allows copies for text and data mining, unless the rights-holder has reserved that use. For content published online, the reservation has to be made in a machine-readable form. This page sets out the wording of the law, the files and signals in which such a reservation is expressed in practice, and why what they said on a particular day is the fact that later has to be shown.

The wording of the law

The rule comes from Article 4 of the Directive on copyright in the Digital Single Market. Paragraph 1 obliges the Member States to provide an exception for reproductions and extractions of lawfully accessible works for the purposes of text and data mining. Paragraph 3 attaches the condition:

“The exception or limitation provided for in paragraph 1 shall apply on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online.”

Directive (EU) 2019/790, Art. 4(3). Source: EUR-Lex, CELEX 32019L0790.

Germany transposed the provision as Section 44b of the Copyright Act (Urheberrechtsgesetz, UrhG). It defines text and data mining in subsection 1, permits the copies in subsection 2 and sets out the reservation in subsection 3:

“(1) ‘Text and data mining’ means the automated analysis of individual or several digital or digitised works for the purpose of gathering information, in particular regarding patterns, trends and correlations.

(2) It is permitted to reproduce lawfully accessible works in order to carry out text and data mining. Copies are to be deleted when they are no longer needed to carry out text and data mining.

(3) Uses in accordance with subsection (2) sentence 1 are permitted only if they have not been reserved by the rightholder. A reservation of use in the case of works which are available online is effective only if it is made in a machine-readable format.”

Section 44b UrhG, in the English translation published by the Federal Ministry of Justice; only the German text is binding. Source: gesetze-im-internet.de.

The AI Act refers to the same reservation. Article 53(1)(c) requires providers of general-purpose AI models to

“put in place a policy to comply with Union law on copyright and related rights, and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790”.

Regulation (EU) 2024/1689, Art. 53(1)(c). Source: EUR-Lex, CELEX 32024R1689.

Where a machine-readable reservation is found in practice

None of these texts names a file format. In practice, reservations aimed at machines are expressed in a handful of places, all of which a machine reads without a human in between:

• /robots.txt: the oldest instruction file on the web. It addresses crawlers by the name they announce and tells them which paths they may fetch.
• /.well-known/tdmrep.json: the TDM Reservation Protocol, drawn up in a W3C Community Group specifically to express a text-and-data-mining reservation.
• HTTP headers and meta tags on the homepage: the same protocol can be declared in a response header or in the HTML head, without a separate file.
• /ai.txt and /llms.txt: newer conventions addressed at AI systems, not standardised.

Which of these forms satisfies the requirement in a particular case is a legal question, and it is not answered here. What can be answered is a question of fact: what a given file or header said on a given day.

Why the day is what has to be shown

A reservation is the state of a file at a point in time. Servers keep no history of it: when robots.txt is edited, the previous version is gone, and nothing on the server says when it changed or what it said before. A question about a reservation, however, usually arises months after the day it concerns. By then, a screenshot taken today shows today, and a server log is the word of one of the parties.

What is needed at that point is a record made on the day itself, by someone who did not know the question would ever be asked.

What this archive records

Since 22 July 2026, two independently operated witnesses fetch these files for around 128,000 EU domains: robots.txt, ai.txt, tdmrep.json and llms.txt every day for every domain, the homepage with its headers and meta tags every day for the core ring and once a week for the broad ring. Each day is sealed into a public root the same night and time-stamped outside this archive, so that nobody, including us, can change it afterwards. What we store lists every address and limit; how it works shows how a third party checks a record without us.

• Whether a domain is already observed, and since when: the coverage check, free of charge.
• What its files said on named days, as a document with instructions to verify it: an evidence extract.
• A domain that should be recorded every day from now on: daily observation.

The archive records what was served. It does not assess whether a reservation is effective, whether a crawler respected it, or who was entitled to make it.