machinewitness

Monthly figures · November 2026

AI reservations in the EU, November 2026

Between 25 October 2026 and 1 November 2026 this archive read the robots.txt of 139,787 most-visited EU websites, by ending or by seat. 102,721 of them served a readable file. 14.4 % of those reserved against at least one AI crawler: they disallowed it completely, or declared ai-train=no.

What is counted. Domains from the Tranco list of the one million most-visited websites worldwide that belong to the EU in one of three ways: they end in .eu or in the country code of one of the 27 EU member states (.de, .fr, .es and the others); they end in a regional ending of the EU (.cat, .berlin, .paris and others); or they end in .com, .org or another generic ending and Wikidata gives their website to an organisation in the EU and to none outside it. These are the most visited sites, not all registered domains of a country. The seat is what Wikidata records, not what this archive asserts.

These figures count. They do not rate anyone, and they say nothing about whether any crawler follows what a file says.

14.4 %reserve against at least one AI crawler
29.0 %name an AI crawler in their robots.txt
13.8 %name one in a file of their own

By ending

Share of domains with a readable robots.txt that reserve against at least one AI crawler. Endings with at least 1,000 readable files; all endings are in the table further down. Hover a bar for the figures.

in the domain's own filethrough a template shared by 50 or more domains
0%5%10%15%20%25%.sk: 23.3 % reserve (18.9 % in their own file, 4.4 % through a shared template), 1,359 readable robots.txt.sk23.3%.cz: 20.9 % reserve (13.4 % in their own file, 7.5 % through a shared template), 3,733 readable robots.txt.cz20.9%.ro: 20.7 % reserve (12.2 % in their own file, 8.5 % through a shared template), 2,857 readable robots.txt.ro20.7%.pl: 19.9 % reserve (15.7 % in their own file, 4.2 % through a shared template), 6,588 readable robots.txt.pl19.9%.gr: 19.1 % reserve (11.5 % in their own file, 7.6 % through a shared template), 2,471 readable robots.txt.gr19.1%.ie: 17.8 % reserve (14.4 % in their own file, 3.4 % through a shared template), 1,000 readable robots.txt.ie17.8%.fi: 17.1 % reserve (13.5 % in their own file, 3.6 % through a shared template), 1,660 readable robots.txt.fi17.1%.it: 16.9 % reserve (11.3 % in their own file, 5.6 % through a shared template), 6,272 readable robots.txt.it16.9%.be: 16.7 % reserve (13.1 % in their own file, 3.5 % through a shared template), 2,186 readable robots.txt.be16.7%.eu: 16.5 % reserve (10.2 % in their own file, 6.2 % through a shared template), 3,329 readable robots.txt.eu16.5%.pt: 16.4 % reserve (12.1 % in their own file, 4.3 % through a shared template), 1,340 readable robots.txt.pt16.4%.fr: 16.2 % reserve (12.5 % in their own file, 3.7 % through a shared template), 7,175 readable robots.txt.fr16.2%.es: 15.8 % reserve (13.0 % in their own file, 2.9 % through a shared template), 3,637 readable robots.txt.es15.8%.hu: 15.8 % reserve (11.3 % in their own file, 4.5 % through a shared template), 2,159 readable robots.txt.hu15.8%.dk: 15.4 % reserve (13.3 % in their own file, 2.0 % through a shared template), 1,784 readable robots.txt.dk15.4%.se: 15.1 % reserve (9.9 % in their own file, 5.2 % through a shared template), 2,780 readable robots.txt.se15.1%.at: 14.8 % reserve (11.6 % in their own file, 3.2 % through a shared template), 2,179 readable robots.txt.at14.8%.de: 12.9 % reserve (10.6 % in their own file, 2.3 % through a shared template), 20,027 readable robots.txt.de12.9%.nl: 6.8 % reserve (5.0 % in their own file, 1.9 % through a shared template), 16,444 readable robots.txt.nl6.8%

Since this issue the figure counts EU by ending or by seat, so it is not compared with the previous one. Counted by the EU country endings only, as before, the share moved from 10.8 % to 14.6 %, a change of +3.8 percentage points. The table below gives the change for every ending.

Most of this rise comes from one change. A ready-made robots.txt template that declares ai-train=no and disallows a list of AI crawlers was served by more than 2,000 domains in this window; in the previous issue, files from templates of any kind carried ai-train=no on a few hundred domains. Individually written files with ai-train=no also became markedly more frequent. The base did not change: the same domains were read, and about as many served a readable file.

Named is not the same as decided

29.0 % of the files name at least one AI crawler. Counted only in files that are not shared with 50 or more other domains, it is 13.8 %. The difference comes from templates: a hosting provider or a shop system writes the same robots.txt for many sites, and some of those templates list AI crawlers, sometimes only to allow them. For reservations the gap is smaller: 10.7 % of all readable files reserve in a file of their own, 3.6 points come through a template.

This month 63 rule sets were shared by 50 or more domains. 38.9 % of all readable files belong to one of them.

In the homepage head

A reservation can also be declared on the homepage itself: as a response header, or as a meta tag in the HTML head. Since 12 October 2026 this archive records the head of every homepage every day (method). In the same window, 91,420 of these websites served a readable homepage.

0.21 %191 domainsdeclare tdm-reservation: 1 (TDM Reservation Protocol), in a header or a meta tag
0.13 %116 domainsdeclare noai or noimageai, in a header or a meta tag
0.32 %288 domainsdeclare at least one of the two

For comparison, 14.4 % of the domains with a readable robots.txt reserve against at least one AI crawler there. The two figures have different bases and are not added up: a domain can declare in both places, in one, or in neither. The TDM Reservation Protocol is a W3C community specification written for the reservation of text and data mining; noai is a widespread convention outside any standard. Both are counted as declared, without judging whether either is legally sufficient. Where a head has no </head> in its first 64 KB, only those 64 KB are read.

llms.txt

/llms.txt is a proposal from 2024 (llmstxt.org): a Markdown file that tells language models what a site contains, beginning with a heading that names the site. It is not a reservation and not a standard. This archive records the path for every domain every day (what that shows). Base here is every counted website, 139,787, because the question is whether a file is there at all.

8.94 %12,497 domainsserve a text file that begins with a Markdown heading, as the proposal requires
9.71 %13,576 domainsserve any non-empty text file under the path
7.97 %11,146 domainsanswer the path with an HTML page, most often their error page, and are not counted

Of the 11,146 HTML answers, 7,636 came after a redirect away from /llms.txt, most often to the homepage; 3,510 were served at the path itself.

Counted is the format, not the content: whether a file is accurate, complete or read by any model is not part of the figure.

The three parts of the figure

Since this issue the figure counts a website as EU by its ending or by the seat of its operator. Both groups beyond the country endings have been observed every day since 18 October 2026. EU regional endings: domains under geographic endings of EU regions and cities: .cat, .eus, .gal, .bzh, .berlin, .brussels, .paris and others. EU seat according to Wikidata: domains under .com, .org, .net and other generic endings whose website Wikidata gives to an organisation in the EU and to none outside it; domains whose website is in fact run by an organisation outside the EU are left out, by hand and with a reason each (list).

PartDomainsReadable robots.txtReservellms.txt
EU country endings128,31593,42814.6 %8.58 %
EU regional endings90164716.5 %4.88 %
EU seat according to Wikidata10,5718,64611.6 %13.64 %
All counted as EU139,787102,72114.4 %8.94 %

Wikidata is the source for the seat, not this archive: where Wikidata is wrong, the group is wrong. Wikidata is also incomplete, so the true number of EU operators under generic endings is larger than this group.

How these figures are made

  1. Which domains. The Tranco list of the million most-visited domains worldwide, filtered to the endings of the 27 EU member states and .eu, to the regional endings of the EU, and to generic endings whose website Wikidata gives to an organisation in the EU and to none outside it. The chart and the table by ending show the country endings.
  2. What is read. The robots.txt of each domain, fetched daily by two independent witnesses and sealed every night (how it works).
  3. Which file counts. The latest robots.txt with status 200 in a seven-day window, www and the bare domain taken together.
  4. What is classified. Whether the file names one of 26 AI crawlers, whether one of them is disallowed completely (Disallow: / in its group), and whether a Content-Signal line declares ai-train=no. A block of all crawlers is not counted, because it is not about AI.
  5. Templates. The file without sitemap, host and comment lines is its rule set. A rule set shared by 50 or more domains counts as a template.
  6. The figure. Shares are of domains with a readable robots.txt. Method version 2; a change of method is announced and never mixed into a series.

Not counted: ai.txt, tdmrep.json and terms of use. The homepage head is a series of its own, described above. What each day holds is described on what we store.

All endings

EndingDomainsReadable robots.txtReserveof which own fileName an AI crawlerai-train=no
.de27,92820,02712.9 %10.6 %25.1 %3.6 %
.nl18,85516,4446.8 %5.0 %65.8 %2.6 %
.fr11,0207,17516.2 %12.5 %21.9 %5.3 %
.it10,1186,27216.9 %11.3 %21.7 %6.9 %
.pl8,9496,58819.9 %15.7 %23.6 %7.5 %
.eu5,6163,32916.5 %10.2 %19.3 %8.5 %
.es5,4023,63715.8 %13.0 %20.8 %5.2 %
.cz4,8913,73320.9 %13.4 %21.6 %11.1 %
.ro3,8352,85720.7 %12.2 %27.9 %9.1 %
.se3,5292,78015.1 %9.9 %29.6 %4.0 %
.gr3,3382,47119.1 %11.5 %22.9 %9.9 %
.be3,1872,18616.7 %13.1 %19.7 %5.2 %
.at2,9572,17914.8 %11.6 %18.7 %3.9 %
.hu2,8922,15915.8 %11.3 %18.7 %7.4 %
.dk2,3271,78415.4 %13.3 %19.0 %4.3 %
.fi2,2211,66017.1 %13.5 %19.4 %4.5 %
.pt1,9851,34016.4 %12.1 %20.8 %6.6 %
.sk1,7981,35923.3 %18.9 %23.3 %11.0 %
.ie1,3981,00017.8 %14.4 %22.3 %6.9 %
.bg1,16088014.8 %10.7 %18.5 %7.4 %
.hr1,07581516.9 %13.4 %19.9 %8.3 %
.lt1,00672318.4 %13.0 %21.6 %10.9 %
.lv84859715.4 %10.4 %17.6 %8.0 %
.si73853514.6 %11.6 %18.7 %6.5 %
.ee72254914.6 %12.6 %17.5 %6.2 %
.lu25517915.6 %14.0 %19.0 %3.9 %
.cy17410323.3 %18.4 %32.0 %11.7 %
.mt916716.4 %10.4 %19.4 %9.0 %

The figures of this issue as data: 2026-11.json.

Using these figures

MachineWitness, “AI reservations in the EU, November 2026”, 1 November 2026, https://machinewitness.eu/blog/ai-reservations-2026-11

Free to quote with that source. For a figure that is not on this page, for one ending, one month or one question of your own, write to contact@machinewitness.eu. We answer in German, English or Spanish.

We do not advise. What a reservation means in law, and what anyone should put in their own files, is a question for a lawyer.

All monthly figures