Reference
What we store about a domain
Written for lawyers and data-protection officers. Every technical term is explained where it first appears.
This page answers one question in full: if MachineWitness observes a domain, what exactly ends up in the archive? Not a summary, the actual fields. We walk through one worked example, beispielfirma.example, from the moment our observer connects to the point where a court could verify the record years later.
Before any of the files below are fetched, we read the domain's robots.txt and obey it. Where it tells us not to fetch a file, we do not fetch it, and the record says so: the observation for that day is recorded as disallowed_by_robots, with the provenance policy: robots_respected, in place of a response. That is a different statement from a server that failed to answer, and the archive keeps the two apart. On 10 August 2026 this applied to roughly 14,000 of the 128,347 domains in the ring, for the three files other than robots.txt itself. We could fetch them anyway (the files are public), and we do not. How a robots.txt is read is itself dated: until 8 September 2026 both witnesses interpreted it with Python's urllib.robotparser, which predates RFC 9309; witness 2 reads it under RFC 9309 (protego) since 9 September 2026, witness 1 since 11 September 2026. The two parsers differ only in edge cases (wildcards, precedence of the longest match); the recorded file is the same either way, only the decision whether to fetch the other files may differ on those days.
Two things are worth saying before the detail. First, we fetch only what any browser can fetch without logging in: a handful of small policy files that a site publishes for machines to read. We do not crawl page content, follow links into applications, or record anything about the people who visit a site. Second, none of this is secret. The value of this archive is not what it holds, which anyone could fetch themselves today; it is that we held it on a specific past date and can prove the record has not been altered since.
1. What we fetch, per domain
Five requests for a domain in our broad ring. Nothing else, ever. Domains in the smaller core ring are fetched at six further addresses since 20 September 2026, listed in the method change below.
| Resource | What it is, and why it matters legally |
|---|---|
| /robots.txt | The oldest machine-readable instruction file on the web. It tells named crawlers which parts of a site they may fetch. Since AI companies publish crawler names (GPTBot, ClaudeBot, Google-Extended and others), this file is where most sites express, or fail to express, a refusal against AI training. |
| /ai.txt | A newer, AI-specific convention for the same purpose. Not yet standardised, which is precisely why its presence or absence on a given date can be contested later. |
| /.well-known/ |
The formal machine-readable reservation of text-and-data-mining rights (TDM Reservation Protocol, W3C). Under Art. 4(3) of the EU DSM Directive a rights-holder's reservation is effective only if machine-readable; the Hamburg Higher Regional Court confirmed in December 2025 that natural-language terms alone do not suffice. This file is the cleanest way to make that reservation. |
| /llms.txt | An emerging convention addressed at large language models directly. |
| / (homepage) | Fetched for its response headers and the first few kilobytes only, never the full page. Some sites express their TDM reservation in an HTTP header or a meta tag rather than a file. |
Each response is capped at a hard byte limit. These files are normally a few kilobytes; the cap exists so that a misconfigured server cannot make us download something large.
Method change, effective 3 August 2026
From that day forward the archive applies the rules below. They are stated here rather than left implicit, because a change in method changes what a record from a given date can be used to prove. Nothing before 3 August 2026 was altered; earlier days keep the method that was in force when they were sealed.
Two rings. The rules differ between a small core ring and a large broad ring. The broad ring is defined by a stated criterion, not by selection: all EU domains within the Tranco top-1M list, as retrieved on 2 August 2026, and since 18 October 2026 two further groups from the same list (method). The core ring is a much smaller set of domains observed more closely. It is assembled by hand and we do not yet publish criteria for it, so it should be read as a working selection rather than a curated register.
- The four opt-out files are still fetched daily for every domain. That is the part that carries legal weight, and it is unchanged.
- The homepage is fetched weekly, not daily, for domains in the broad ring. The honest consequence: where a site expresses its reservation only in a homepage meta tag or HTTP header, our broad-ring record of that reservation is precise to the week, not to the day. For the core ring nothing changes: there the homepage is still fetched daily.
- Byte caps apply to files, not to the homepage. Since 20 September 2026 the homepage is stored whole in both rings; what separates them is how often it is fetched, not how much of it survives. A cap remains where a file can run to any length: 256 KB on the machine-readable files in the broad ring, 5 MB on the sitemap in the core ring. Anything beyond is marked
truncated, so a fragment is never presented as a whole. One operating limit applies to everything: a single retrieval larger than 32 MB is cut off at that size and markedtruncatedin the same way. It concerns media files, faulty servers and endless streams, not documents, and it does not change what the record proves for what it holds. - Error responses are recorded without their body. Status code, all headers, size and content hash are kept, so the fact and shape of the error remain provable; the error page itself is not stored. One exception:
403 Forbiddenbodies are kept up to 32 KB, because a refusal aimed at crawlers is sometimes stated in that page.
Method change, effective 17 September 2026
From that day forward the record grows in three places. As before, nothing earlier was altered: days before 17 September 2026 keep the method in force when they were sealed. The first crawl under the new scope began on 17 September 2026 at 03:00 UTC.
The core ring now has a stated criterion. Until 16 September 2026 it was a working selection of 115 domains, assembled by hand. From 17 September it holds 1,133 domains, composed as follows. 1,057 domains of organisations in the EU that publish, as recorded in Wikidata with a live website, whether newspapers and news sites, broadcasters, book publishers, news and picture agencies or collecting societies, and whose site at the same time carried a signal addressed at AI crawlers that we had already observed: a rule for a named AI bot in robots.txt or a Content-Signal header. They sit in 26 member states. Added to them: 43 further Wikidata-recorded publishers that were in the ring before and carry no such signal; 14 domains kept from the earlier selection for a stated reason; 4 domains that carry such a signal but have no Wikidata entry; and the 15 domains of the operator's own company, kept as a control series rather than hidden. For every core domain the record now names the organisation, its country and its Wikidata identifier. Where Wikidata attributes a domain to more than one organisation, the shortest label was taken and the entry is marked as unsettled (61 domains), to be corrected as curation proceeds; the domain itself is unaffected. The core-ring rules of 3 August 2026, homepage daily and no byte cap, apply to all of them from 17 September. The broad ring is unchanged. One domain of the operator's own company that no longer resolves in the DNS was withdrawn from the ring on the same day, and three others were added; the ring manifest of that date lists every domain with its ring.
Operator declarations. For each of 13 crawler operators (AI2, Amazon, Anthropic, Apple, Cohere, Common Crawl, Diffbot, DuckDuckGo, Google, Microsoft, Mistral, OpenAI, Perplexity) the declarations the operator itself publishes about its crawlers, the documentation page and the published IP ranges (JSON), are fetched daily, without byte cap, and hashed, sealed and anchored like every other artefact: 26 addresses. We do not check whether any access actually came from those ranges; what is witnessed is solely what the operator declared on day T. Where a redirect was in place, the target is recorded, so that the provenance carries the document rather than the redirect. Left out, with the reason stated: pages that serve nothing to a client without a browser (Meta's crawler documentation), because an empty shell proves nothing. The day's state of this stream is issued as an evidence extract of the operator's domain, like any other, at the published fee.
Provider declarations. By the same mechanism, the published usage policies, terms and crawler policies of 12 of these operators (AI2, Amazon, Anthropic, Apple, Cohere, Common Crawl, Diffbot, DuckDuckGo, Google, Meta, Microsoft, Mistral): 32 addresses, daily, without byte cap. Two operators, OpenAI and Perplexity, do not serve these pages to a client that is not a browser; their terms are therefore not in the record, and we say so here rather than fill the gap by other means. Together these are the counterpart to the opt-out files: what a site had reserved on day T, and what the operator had declared about its own conduct on the same day. Nothing is evaluated: the record holds the text, not a judgement on it. Like the operator declarations above, the day's state of this stream is issued as an evidence extract of the operator's domain, at the published fee.
Method change, effective 20 September 2026
From that day forward the core ring is observed at six further addresses per domain. As before, nothing earlier was altered: days before 20 September 2026 keep the method in force when they were sealed. The broad ring is unchanged and still receives five requests per domain, nothing else.
What was added, and why now. Two kinds of file that address machines rather than readers. The first states a price rather than a refusal: /.well-known/rsl.xml and /rsl.xml, the licence document of the Really Simple Licensing specification, fetched at both locations because the specification names the first while the adopters we observe point their robots.txt at the second. The second kind is what a site declares to autonomous agents: the A2A Agent Card (/.well-known/agent-card.json), the older plugin manifest (/.well-known/ai-plugin.json), and the key directory used to authenticate signed bot requests (/.well-known/http-message-signatures-directory). Added alongside them: /sitemap.xml, the file itself and never the addresses listed in it, which can show that a page already existed on a given date.
Adoption on the day we began was almost nil, and that is stated rather than hidden. Measured across the core ring on 19 and 20 September 2026: two domains served an RSL licence line, and exactly one domain of 1,119 served any of the agent files. We did not wait for that to change. A record that begins after a practice becomes common cannot show when it began, and an absent file is itself a dated statement: on day T, this domain declared nothing to agents. The cost of holding that statement is close to nothing, because error responses are recorded without their body.
One byte cap, and one deliberate omission. The core ring remains uncapped, with a single exception: the sitemap is capped at 5 MB and anything beyond is marked truncated. It is the only one of these files that can run to megabytes, and what it is used to show rests on the entries it contains, not on the list being complete. Deliberately not added: the discovery paths proposed for the Model Context Protocol. Two competing paths exist and neither had been adopted into that specification at the time of writing, so recording one of them daily would build an unbroken series about an address that may never be the address. When a path is settled, adding it will be a further method change, stated here like this one.
Method change, effective 29 September 2026
From that day forward each homepage in the broad ring has a fixed weekday of its own. The promise does not change: the homepage is fetched at least once a week, the four opt-out files every day. What changes is only which day of the week. Witness 1 applies the rule from its crawl of 29 September 2026, witness 2 from its crawl of 30 September 2026. Earlier days keep the method in force when they were sealed.
The rule, so that anyone can recompute it. Take the host name of the homepage address in lower case, without port, and compute its SHA-256. Read the digest as one unsigned big-endian integer, add the witness offset (0 for witness 1, 3 for witness 2) and take the remainder modulo 7: that is the domain's slot. A day belongs to slot n when the number of days since 1 January 1970 leaves the remainder n modulo 7. A broad-ring homepage is fetched on a given day if it has never been stored, if its last stored copy is seven or more days old, or if the day belongs to its slot and it was not already stored that day. A domain unreachable on its day is therefore tried again on each following day until a copy is stored, and then returns to its slot.
Why. Until then a homepage fell due seven days after its last stored copy. Because the broad ring was added in batches, whole batches fell due on the same weekday: two nights of the week carried almost all broad-ring homepages and five carried almost none. The rule spreads them evenly. The offsets differ between the witnesses on purpose: the two witnesses fetch a given homepage on different days, so no single day's failure can remove it from both, and each homepage is seen twice a week, once by each witness, on different days. The consequence is stated openly: for a broad-ring homepage the two witnesses do not, as a rule, hold a copy from the same day. What both witnesses hold for the same day is the four opt-out files, fetched daily by each.
The changeover. No gap grows beyond seven days through the change. In the first week some homepages are fetched earlier than seven days after their previous copy, once, because their new slot comes round sooner; after that each domain is fetched on its slot.
Method change, effective 5 October 2026
From that day forward a sealed day can hold one further fingerprint beside the observations: the declarations of the AI Yes/No list that people confirmed or changed since the previous seal. It is not an observation. It records what people declared to us, not what a domain served, and its class says so: its address is https://machinewitness.eu/liste/erklaert and its status erklaert. A day without such declarations has no such fingerprint and is built exactly as before.
The fingerprint, so that anyone can recompute it. It has the same form as every other fingerprint of the day. What it covers is a file with that day's declarations, sorted, each one sha256("mwdecl-v1|" key "|" kind "|" platform "|" target "|" choice "|" day), where the key is a random value that only the person received, on their management page. The declarations are combined pairwise like the observations, and the resulting root is part of the fingerprint's metadata. Without the key a declaration names no one; with it, the person can prove an entry from the receipt to the published daily root, also after deleting it. The check page is /liste/pruefen.
Why it is kept apart. Observations and declarations answer different questions. One fingerprint of its own class leaves everything about the observations unchanged, and every number of observations this archive publishes excludes it.
Method change, effective 12 October 2026
From that day forward the head of every broad-ring homepage is recorded every day, as an artefact class of its own. Witness 1 applies the rule from its crawl of 12 October 2026. Witness 2 applies it from its crawl of 13 October 2026. Earlier days keep the method in force when they were sealed.
What is recorded. For every broad-ring domain, one further daily request to the homepage address that stops reading at the end of the HTML head: the bytes up to and including the first </head>, matched without regard to case. Where none arrives within the first 64 KB, those 64 KB are kept and marked truncated. The cut is purely syntactic, so anyone can redo it on a stored page with a single search. All response headers are recorded as with every observation. The address is the homepage address; what tells the class apart is its provenance, which is part of the fingerprint: artifact: homepage-head and head_cut (head_end, fallback or eof).
Why. Since 3 August 2026 the full homepage is fetched weekly in the broad ring, and a reservation stated only in a meta tag or a response header was therefore recorded to the week, not to the day. Both now sit in a daily record for every domain. The full homepage stays weekly in the broad ring and daily in the core ring; in the core ring the head is part of the full page and is not fetched separately.
What it follows, and what it is not counted as. The head keeps no state of its own and follows its homepage: a robots.txt that excludes the homepage excludes its head, and a homepage paused after repeated failures, or stopped on request, pauses its head. It is holding, not a signal. About half of all heads differ from one day to the next by values that change on every delivery (one-time tokens, build marks), so a head is never counted as a change, and no change figure this archive publishes includes it.
Method change, effective 18 October 2026
From that day forward the broad ring also includes two groups of domains that the country endings miss. Both witnesses apply it from their crawl of 18 October 2026. Earlier days keep the method in force when they were sealed.
What is added. EU regional endings: every domain in the Tranco top-1M list, as retrieved on 17 October 2026, under a geographic ending of a region or city in the EU: .cat, .eus, .gal, .bzh, .corsica, .alsace, .paris, .berlin, .hamburg, .bayern, .koeln, .cologne, .nrw, .ruhr, .saarland, .wien, .tirol, .brussels, .vlaanderen, .gent, .amsterdam, .frl, .barcelona, .madrid and .bcn (901 domains). EU seat according to Wikidata: every domain in the same list under any other ending whose website Wikidata (property P856) gives to an entity with a country (P17) or a headquarters (P159) in the EU, and to no entity outside the EU (10,605 domains). Domains already observed were not added again. Both lists are kept as dated input files of the archive.
Why. The country ending is a stated criterion, but it is not the seat of the operator: many publishers and companies in the EU use .com, .org or .net. Registration data cannot answer the question, because for .com the holder is usually withheld, and neither can the location of a server. Wikidata can, as a public and citable source. Where Wikidata is wrong, the group is wrong; the archive does not claim a seat of its own.
What applies to them. The same rules as for the rest of the broad ring. From the monthly figures of November 2026 on, both groups count as EU together with the country endings; the figures by country ending are still shown on their own. Domains whose website is in fact run by an organisation outside the EU are left out of the figures, not of the observation (list with reasons).
2. What one observation contains
An observation is one recording of one resource at one moment. Here is a complete one for our example domain, in the form the archive holds it.
The content itself
GET https://beispielfirma.example/robots.txt → HTTP 200, 412 bytes User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
The exact bytes the server delivered, stored unchanged and compressed. Not our summary of them, not a rendering: the bytes. In a dispute the question is never "what does the file say" but "what did it say on 14 August 2027".
The fingerprint
payload_sha256 = 9f2a41d7c8e05b3a…7d1e6084bb93c2f5 payload_size = 412 bytes
A SHA-256 hash is a short fixed-length code computed from the exact bytes of a file. Change a single character and the code changes completely, and it is computationally infeasible to construct a different file with the same code. It functions as a fingerprint of one specific version of a document. This is also the filing number: the content is stored under its own fingerprint, so nothing can be substituted without the label ceasing to match.
The circumstances of delivery
observed_at = 2027-08-14 03:00:11 UTC
observer_id = witness-1
status_code = 200
final_url = https://www.beispielfirma.example/robots.txt (after redirect)
http_version = HTTP/1.1
headers = content-type: text/plain; charset=UTF-8
cache-control: max-age=600
x-served-by: cache-fra-…
(complete response headers, verbatim)
tls_version = TLSv1.3
peer_cert = subject: *.beispielfirma.example
issuer: GlobalSign Atlas R3 DV TLS CA
valid: 2027-06-01 → 2028-06-01
sha256: 3b8c07f2ad91…e5240ac71f6b88
(fingerprint of the server certificate,
plus the five parsed fields above)
peer_cert_chain = 3b8c07f2ad91…e5240ac71f6b88 (server certificate)
6f1d0ba47c33…9a7e1c40db2f51 (intermediate)
c04e83fa1d67…2b95e7708aa361 (intermediate)
(the chain as the server delivered it, in that
order; every certificate stored in full)
observer_addr = the IP address our observer connected from
peer_addr = the IP address that answered
This is what lawyers call chain of custody: not merely "this file existed" but "this server, presenting this TLS certificate, delivered these bytes to this observer at this moment". The TLS certificate matters because it ties the delivery to a party who could obtain a certificate for that domain. All of it is sealed together with the content, so no part can be swapped later.
Precisely what is recorded here, and what is not. Since 15 September 2026 we store the full certificate chain as the server delivers it: the server certificate and the intermediates sent alongside it, each kept in full, with the fingerprints recorded in the order they arrived. It is not a chain rebuilt against our own trust store: a witness records what was delivered, and whether that chain validates is the examiner's question, not ours. Alongside it we continue to record the SHA-256 fingerprint of the server certificate and the five parsed fields shown above.
The change does not work backwards. For observations before 15 September 2026 only the fingerprint exists, not the certificate; where that certificate is needed, it can generally be retrieved independently from the public Certificate Transparency logs (RFC 6962), the same audit infrastructure browsers rely on. Older records keep the state they had at the time, and nothing is filled in afterwards. This settles the undertaking this page has carried since 1 August 2026.
The two addresses. observer_addr and peer_addr are meant to hold the IP address and port our witness connected from and the address that answered. Until 24 September 2026 both fields were recorded empty in every observation: our software asked its network library for the addresses under names that library does not use. We noticed while preparing the first delivery document and corrected it for 25 September 2026. That correction was incomplete: under the parallel load of the daily crawl the addresses were read only after the connection had already been handed back, so from 25 to 29 September 2026 only about one observation in twenty carries them (single captures, such as the point-in-time record, carry both). Corrected again for 30 September 2026; from that day on every retrieval that received an answer carries both addresses. Earlier records keep the state they had, and nothing is filled in afterwards.
The change flag
changed = true (fingerprint differs from the previous observation)
Set when today's fingerprint differs from yesterday's for the same resource. It is the reason this archive exists: these files change silently, without a public history, and the change itself is often the disputed fact.
3. What an absent file looks like
A record of "nothing was there" is evidence too, and it is recorded with the same care as content. If our example domain serves no ai.txt:
GET https://beispielfirma.example/ai.txt → HTTP 404, no payload stored status, complete headers, payload fingerprint and TLS details recorded as above
This distinction carries real weight. "The site had published no machine-readable AI reservation on that date" is frequently the decisive fact in a TDM dispute, and it can only be shown by someone who looked and wrote down that they found nothing. Two further cases are recorded just as explicitly: a server that refuses our observer (HTTP 403) produces a record of the refusal, not of content; and a resource that a site's own robots.txt tells us not to fetch is recorded as deliberately not fetched. We honour that instruction and log that we did.
4. How the record is sealed
An archive kept by an interested party proves little on its own: we could, in principle, rewrite our own database. Four steps address that objection, each independent of the others.
Every observation of the day is bound into one figure
All fingerprints recorded that day are combined pairwise, level upon level, until a single daily root remains that depends on every record beneath it. Change any one observation afterwards and the root no longer matches. This is the construction used by the public logs that browsers rely on to audit TLS certificates (RFC 6962).
That figure is deposited outside our control
The daily root is submitted the same day to OpenTimestamps, which aggregates it towards the Bitcoin blockchain, and separately to an independent RFC 3161 time-stamping authority, which returns a signed token immediately. Since 31 July 2026 a qualified eIDAS time-stamp is obtained as well. We cannot rewrite any of them. This is what upgrades the seal from an internal claim to external proof: it shows the record existed no later than that day.
Status of the Bitcoin anchor: completed on 7 August 2026. The two time-stamping authorities return their proof at once, and those tokens are held here. OpenTimestamps works in two stages: the root is submitted immediately, but the resulting receipt only becomes self-contained once the Bitcoin confirmation is fetched back into it. Until 7 August 2026 we had not performed that second step, and an earlier version of this passage said so. It has now been carried out for every sealed day on both witnesses, and a daily job performs it from here on, so the receipts stored here reference Bitcoin block headers rather than calendar servers. The receipts were changed by this step, but what they attest was not: the hash each receipt witnesses was recorded before and after and is unchanged for every day. The Bitcoin path no longer depends on those calendar servers remaining reachable. The RFC 3161 and qualified eIDAS anchors were unaffected throughout and stand on their own.
The root is published
Each daily root appears in our public root log, with stable URLs and machine-readable form. Anyone may keep their own copy the day it appears; several third-party systems capture it automatically.
A second, independent witness observes the same targets
A separate system, on different infrastructure in another country and with a different operator key, records independently and publishes its own daily root. Neither machine holds credentials for the other and neither can write to the other's database. Two accounts kept independently are harder to dismiss than one witness repeating himself.
Two limits stated plainly. First, this began on 4 August 2026: every day sealed from 22 July to 3 August 2026 rests on the first witness alone, and the public log shows that gap rather than hiding it. Second, the two daily roots are never identical, by construction: each witness crawls on its own schedule, so the set of observations behind each root differs. A comparison is therefore made observation by observation, not by checking whether two roots match; identical roots would in fact indicate that the two systems were not independent.
5. What an evidence extract contains
The archive is not browsable, by design. What we produce on request, for a specific domain and date range, is an evidence extract: a package containing
- the stored payload for each observation in the range, byte for byte;
- the complete sealed circumstances of delivery, as set out in section 2;
- the fingerprint of each observation;
- an inclusion proof: a short mathematical receipt showing that this specific observation is contained in that day's root, checkable without trusting us and without access to the rest of the archive;
- the daily root and the external timestamp receipts;
- step-by-step verification instructions that any competent technical expert can follow with standard tools.
The point of the last item is worth stating plainly: the extract is designed so that the opposing party's expert can verify it. A record only one side can check is not evidence.
How one is requested, the terms it is issued under, and what it costs are set out on the evidence extract page. Whether this archive observes a given domain at all can be checked first, at no cost, with the coverage check.
6. If someone asks us to erase
Most of what we hold contains no personal data: these are technical policy files published by organisations. Where a legitimate erasure request under Art. 17 GDPR does concern archived content, we perform what we call a tombstone: the stored content is deleted and replaced by a marker. Deleted means deleted, and no backup restores it.
What remains is the fingerprint, the seal, and the proofs. The consequence is worth understanding precisely: afterwards, nobody can learn what the file said, but it remains provable that a file with exactly this fingerprint stood at this address on this date, and that the surrounding records of that day are untouched. Retaining the fingerprint rests on Art. 17(3)(e) GDPR, because deleting it would break the proof chain for every other observation sealed that day, including those belonging to unrelated third parties.
On request we also exclude a domain from all future observation, with or without an erasure. Both are free, and neither needs a reason. For an exclusion we do ask you to show that you control the domain: a DNS TXT record, or a file at a path we name, whichever is easier. Not because the request needs justifying, but because an exclusion asked for by someone else would take a domain out of the record without its operator ever learning of it. An erasure under Art. 17 GDPR carries no such step: there we ask for identification only where there is real doubt. See Privacy and Crawler & contact.
7. What this record does not prove
Stating the limits precisely is part of being a credible witness.
- It shows what a server delivered to our observer, at the recorded moments. It says nothing about the intervals between observations, and nothing about what other visitors saw.
- It shows content and circumstances of delivery, not authorship, intent, or lawfulness. Whether a reservation was effective, whether a crawler respected it, and what follows legally are questions for counsel and the court.
- External anchoring proves a record existed no later than its anchor date. It proves nothing about any earlier date.
- A missing file proves that the resource was absent at that address at that moment, not that the operator never reserved rights by some other means.
- We record public, machine-readable statements by organisations. We observe no private data, no user behaviour, and no individuals.
We are the flight recorder, not the investigator. What the record means is for others to argue.