Reference

What we store about a domain

Written for lawyers and data-protection officers. Every technical term is explained where it first appears.

This page answers one question in full: if MachineWitness observes a domain, what exactly ends up in the archive? Not a summary, the actual fields. We walk through one worked example, beispielfirma.example, from the moment our observer connects to the point where a court could verify the record years later.

Before any of the files below are fetched, we read the domain's robots.txt and obey it. Where it tells us not to fetch a file, we do not fetch it, and the record says so: the observation for that day is recorded as disallowed_by_robots, with the provenance policy: robots_respected, in place of a response. That is a different statement from a server that failed to answer, and the archive keeps the two apart. On 10 August 2026 this applied to roughly 14,000 of the 128,347 domains in the ring, for the three files other than robots.txt itself. We could fetch them anyway (the files are public), and we do not. How a robots.txt is read is itself dated: until 8 September 2026 both witnesses interpreted it with Python's urllib.robotparser, which predates RFC 9309; witness 2 reads it under RFC 9309 (protego) since 9 September 2026, witness 1 since 11 September 2026. The two parsers differ only in edge cases (wildcards, precedence of the longest match); the recorded file is the same either way, only the decision whether to fetch the other files may differ on those days.

Two things are worth saying before the detail. First, we fetch only what any browser can fetch without logging in: a handful of small policy files that a site publishes for machines to read. We do not crawl page content, follow links into applications, or record anything about the people who visit a site. Second, none of this is secret. The value of this archive is not what it holds, which anyone could fetch themselves today; it is that we held it on a specific past date and can prove the record has not been altered since.

1. What we fetch, per domain

Five requests. Nothing else, ever, for a domain in our broad ring.

Resource	What it is, and why it matters legally
/robots.txt	The oldest machine-readable instruction file on the web. It tells named crawlers which parts of a site they may fetch. Since AI companies publish crawler names (GPTBot, ClaudeBot, Google-Extended and others), this file is where most sites express, or fail to express, a refusal against AI training.
/ai.txt	A newer, AI-specific convention for the same purpose. Not yet standardised, which is precisely why its presence or absence on a given date can be contested later.
/.well-known/tdmrep.json	The formal machine-readable reservation of text-and-data-mining rights (TDM Reservation Protocol, W3C). Under Art. 4(3) of the EU DSM Directive a rights-holder's reservation is effective only if machine-readable; the Hamburg Higher Regional Court confirmed in December 2025 that natural-language terms alone do not suffice. This file is the cleanest way to make that reservation.
/llms.txt	An emerging convention addressed at large language models directly.
/ (homepage)	Fetched for its response headers and the first few kilobytes only, never the full page. Some sites express their TDM reservation in an HTTP header or a meta tag rather than a file.

Each response is capped at a hard byte limit. These files are normally a few kilobytes; the cap exists so that a misconfigured server cannot make us download something large.

Method change, effective 3 August 2026

From that day forward the archive applies the rules below. They are stated here rather than left implicit, because a change in method changes what a record from a given date can be used to prove. Nothing before 3 August 2026 was altered; earlier days keep the method that was in force when they were sealed.

Two rings. The rules differ between a small core ring and a large broad ring. The broad ring is defined by a stated criterion, not by selection: all EU domains within the Tranco top-1M list, as retrieved on 2 August 2026. The core ring is a much smaller set of domains observed more closely. It is assembled by hand and we do not yet publish criteria for it, so it should be read as a working selection rather than a curated register.

The four opt-out files are still fetched daily for every domain. That is the part that carries legal weight, and it is unchanged.
The homepage is fetched weekly, not daily, for domains in the broad ring. The honest consequence: where a site expresses its reservation only in a homepage meta tag or HTTP header, our broad-ring record of that reservation is precise to the week, not to the day. For the core ring nothing changes: there the homepage is still fetched daily.
Byte caps differ by ring. Core ring: uncapped. Broad ring: 256 KB, with anything beyond that marked truncated. What is kept is always the beginning of the response, where these signals live.
Error responses are recorded without their body. Status code, all headers, size and content hash are kept, so the fact and shape of the error remain provable; the error page itself is not stored. One exception: 403 Forbidden bodies are kept up to 32 KB, because a refusal aimed at crawlers is sometimes stated in that page.

Method change, effective 17 September 2026

From that day forward the record grows in three places. As before, nothing earlier was altered: days before 17 September 2026 keep the method in force when they were sealed. The first crawl under the new scope began on 17 September 2026 at 03:00 UTC.

The core ring now has a stated criterion. Until 16 September 2026 it was a working selection of 115 domains, assembled by hand. From 17 September it holds 1,133 domains, composed as follows. 1,057 domains of organisations in the EU that publish, as recorded in Wikidata with a live website, whether newspapers and news sites, broadcasters, book publishers, news and picture agencies or collecting societies, and whose site at the same time carried a signal addressed at AI crawlers that we had already observed: a rule for a named AI bot in robots.txt or a Content-Signal header. They sit in 26 member states. Added to them: 43 further Wikidata-recorded publishers that were in the ring before and carry no such signal; 14 domains kept from the earlier selection for a stated reason; 4 domains that carry such a signal but have no Wikidata entry; and the 15 domains of the operator's own company, kept as a control series rather than hidden. For every core domain the record now names the organisation, its country and its Wikidata identifier. Where Wikidata attributes a domain to more than one organisation, the shortest label was taken and the entry is marked as unsettled (61 domains), to be corrected as curation proceeds; the domain itself is unaffected. The core-ring rules of 3 August 2026, homepage daily and no byte cap, apply to all of them from 17 September. The broad ring is unchanged. One domain of the operator's own company that no longer resolves in the DNS was withdrawn from the ring on the same day, and three others were added; the ring manifest of that date lists every domain with its ring.

Operator declarations. For each of 13 crawler operators (AI2, Amazon, Anthropic, Apple, Cohere, Common Crawl, Diffbot, DuckDuckGo, Google, Microsoft, Mistral, OpenAI, Perplexity) the declarations the operator itself publishes about its crawlers, the documentation page and the published IP ranges (JSON), are fetched daily, without byte cap, and hashed, sealed and anchored like every other artefact: 26 addresses. We do not check whether any access actually came from those ranges; what is witnessed is solely what the operator declared on day T. Where a redirect was in place, the target is recorded, so that the provenance carries the document rather than the redirect. Left out, with the reason stated: pages that serve nothing to a client without a browser (Meta's crawler documentation), because an empty shell proves nothing. The day's state of this stream can be ordered as a further kind of file together with an evidence extract.

Provider declarations. By the same mechanism, the published usage policies, terms and crawler policies of 12 of these operators (AI2, Amazon, Anthropic, Apple, Cohere, Common Crawl, Diffbot, DuckDuckGo, Google, Meta, Microsoft, Mistral): 32 addresses, daily, without byte cap. Two operators, OpenAI and Perplexity, do not serve these pages to a client that is not a browser; their terms are therefore not in the record, and we say so here rather than fill the gap by other means. Together these are the counterpart to the opt-out files: what a site had reserved on day T, and what the operator had declared about its own conduct on the same day. Nothing is evaluated: the record holds the text, not a judgement on it. Like the operator declarations above, the day's state of this stream can be ordered as a further kind of file together with an evidence extract.

2. What one observation contains

An observation is one recording of one resource at one moment. Here is a complete one for our example domain, in the form the archive holds it.

The content itself

GET https://beispielfirma.example/robots.txt
→ HTTP 200, 412 bytes

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The exact bytes the server delivered, stored unchanged and compressed. Not our summary of them, not a rendering: the bytes. In a dispute the question is never "what does the file say" but "what did it say on 14 August 2027".

The fingerprint

payload_sha256 = 9f2a41d7c8e05b3a…7d1e6084bb93c2f5
payload_size   = 412 bytes

A SHA-256 hash is a short fixed-length code computed from the exact bytes of a file. Change a single character and the code changes completely, and it is computationally infeasible to construct a different file with the same code. It functions as a fingerprint of one specific version of a document. This is also the filing number: the content is stored under its own fingerprint, so nothing can be substituted without the label ceasing to match.

The circumstances of delivery

observed_at   = 2027-08-14 03:00:11 UTC
observer_id   = witness-1
status_code   = 200
final_url     = https://www.beispielfirma.example/robots.txt   (after redirect)
http_version  = HTTP/1.1
headers       = content-type: text/plain; charset=UTF-8
                cache-control: max-age=600
                x-served-by: cache-fra-…
                (complete response headers, verbatim)
tls_version   = TLSv1.3
peer_cert     = subject:  *.beispielfirma.example
                issuer:   GlobalSign Atlas R3 DV TLS CA
                valid:    2027-06-01 → 2028-06-01
                sha256:   3b8c07f2ad91…e5240ac71f6b88
                (fingerprint of the server certificate,
                 plus the five parsed fields above)
peer_cert_chain = 3b8c07f2ad91…e5240ac71f6b88   (server certificate)
                6f1d0ba47c33…9a7e1c40db2f51   (intermediate)
                c04e83fa1d67…2b95e7708aa361   (intermediate)
                (the chain as the server delivered it, in that
                 order; every certificate stored in full)
observer_addr = the IP address our observer connected from
peer_addr     = the IP address that answered

This is what lawyers call chain of custody: not merely "this file existed" but "this server, presenting this TLS certificate, delivered these bytes to this observer at this moment". The TLS certificate matters because it ties the delivery to a party who could obtain a certificate for that domain. All of it is sealed together with the content, so no part can be swapped later.

Precisely what is recorded here, and what is not. Since 15 September 2026 we store the full certificate chain as the server delivers it: the server certificate and the intermediates sent alongside it, each kept in full, with the fingerprints recorded in the order they arrived. It is not a chain rebuilt against our own trust store: a witness records what was delivered, and whether that chain validates is the examiner's question, not ours. Alongside it we continue to record the SHA-256 fingerprint of the server certificate and the five parsed fields shown above.

The change does not work backwards. For observations before 15 September 2026 only the fingerprint exists, not the certificate; where that certificate is needed, it can generally be retrieved independently from the public Certificate Transparency logs (RFC 6962), the same audit infrastructure browsers rely on. Older records keep the state they had at the time, and nothing is filled in afterwards. This settles the undertaking this page has carried since 1 August 2026.

The change flag

changed = true   (fingerprint differs from the previous observation)

Set when today's fingerprint differs from yesterday's for the same resource. It is the reason this archive exists: these files change silently, without a public history, and the change itself is often the disputed fact.

3. What an absent file looks like

A record of "nothing was there" is evidence too, and it is recorded with the same care as content. If our example domain serves no ai.txt:

GET https://beispielfirma.example/ai.txt
→ HTTP 404, no payload stored
   status, complete headers, payload fingerprint and
   TLS details recorded as above

This distinction carries real weight. "The site had published no machine-readable AI reservation on that date" is frequently the decisive fact in a TDM dispute, and it can only be shown by someone who looked and wrote down that they found nothing. Two further cases are recorded just as explicitly: a server that refuses our observer (HTTP 403) produces a record of the refusal, not of content; and a resource that a site's own robots.txt tells us not to fetch is recorded as deliberately not fetched. We honour that instruction and log that we did.

4. How the record is sealed

An archive kept by an interested party proves little on its own: we could, in principle, rewrite our own database. Four steps address that objection, each independent of the others.

Step 1 · same day

Every observation of the day is bound into one figure

All fingerprints recorded that day are combined pairwise, level upon level, until a single daily root remains that depends on every record beneath it. Change any one observation afterwards and the root no longer matches. This is the construction used by the public logs that browsers rely on to audit TLS certificates (RFC 6962).

Step 2 · submitted same day

That figure is deposited outside our control

The daily root is submitted the same day to OpenTimestamps, which aggregates it towards the Bitcoin blockchain, and separately to an independent RFC 3161 time-stamping authority, which returns a signed token immediately. Since 31 July 2026 a qualified eIDAS time-stamp is obtained as well. We cannot rewrite any of them. This is what upgrades the seal from an internal claim to external proof: it shows the record existed no later than that day.

Status of the Bitcoin anchor: completed on 7 August 2026. The two time-stamping authorities return their proof at once, and those tokens are held here. OpenTimestamps works in two stages: the root is submitted immediately, but the resulting receipt only becomes self-contained once the Bitcoin confirmation is fetched back into it. Until 7 August 2026 we had not performed that second step, and an earlier version of this passage said so. It has now been carried out for every sealed day on both witnesses, and a daily job performs it from here on, so the receipts stored here reference Bitcoin block headers rather than calendar servers. The receipts were changed by this step, but what they attest was not: the hash each receipt witnesses was recorded before and after and is unchanged for every day. The Bitcoin path no longer depends on those calendar servers remaining reachable. The RFC 3161 and qualified eIDAS anchors were unaffected throughout and stand on their own.

Step 3 · same day

The root is published

Each daily root appears in our public root log, with stable URLs and machine-readable form. Anyone may keep their own copy the day it appears; several third-party systems capture it automatically.

Step 4 · operating since 2026-08-04

A second, independent witness observes the same targets

A separate system, on different infrastructure in another country and with a different operator key, records independently and publishes its own daily root. Neither machine holds credentials for the other and neither can write to the other's database. Two accounts kept independently are harder to dismiss than one witness repeating himself.

Two limits stated plainly. First, this began on 4 August 2026: every day sealed from 22 July to 3 August 2026 rests on the first witness alone, and the public log shows that gap rather than hiding it. Second, the two daily roots are never identical, by construction: each witness crawls on its own schedule, so the set of observations behind each root differs. A comparison is therefore made observation by observation, not by checking whether two roots match; identical roots would in fact indicate that the two systems were not independent.

5. What an evidence extract contains

The archive is not browsable, by design. What we produce on request, for a specific domain and date range, is an evidence extract: a package containing

the stored payload for each observation in the range, byte for byte;
the complete sealed circumstances of delivery, as set out in section 2;
the fingerprint of each observation;
an inclusion proof: a short mathematical receipt showing that this specific observation is contained in that day's root, checkable without trusting us and without access to the rest of the archive;
the daily root and the external timestamp receipts;
step-by-step verification instructions that any competent technical expert can follow with standard tools.

The point of the last item is worth stating plainly: the extract is designed so that the opposing party's expert can verify it. A record only one side can check is not evidence.

How one is requested, the terms it is issued under, and what it costs are set out on the evidence extract page. Whether this archive observes a given domain at all can be checked first, at no cost, with the coverage check.

6. If someone asks us to erase

Most of what we hold contains no personal data: these are technical policy files published by organisations. Where a legitimate erasure request under Art. 17 GDPR does concern archived content, we perform what we call a tombstone: the stored content is deleted and replaced by a marker. Deleted means deleted, and no backup restores it.

What remains is the fingerprint, the seal, and the proofs. The consequence is worth understanding precisely: afterwards, nobody can learn what the file said, but it remains provable that a file with exactly this fingerprint stood at this address on this date, and that the surrounding records of that day are untouched. Retaining the fingerprint rests on Art. 17(3)(e) GDPR, because deleting it would break the proof chain for every other observation sealed that day, including those belonging to unrelated third parties.

On request we also exclude a domain from all future observation, with or without an erasure. Both are free, and neither needs a reason. For an exclusion we do ask you to show that you control the domain: a DNS TXT record, or a file at a path we name, whichever is easier. Not because the request needs justifying, but because an exclusion asked for by someone else would take a domain out of the record without its operator ever learning of it. An erasure under Art. 17 GDPR carries no such step: there we ask for identification only where there is real doubt. See Privacy and Crawler & contact.

7. What this record does not prove

Stating the limits precisely is part of being a credible witness.

It shows what a server delivered to our observer, at the recorded moments. It says nothing about the intervals between observations, and nothing about what other visitors saw.
It shows content and circumstances of delivery, not authorship, intent, or lawfulness. Whether a reservation was effective, whether a crawler respected it, and what follows legally are questions for counsel and the court.
External anchoring proves a record existed no later than its anchor date. It proves nothing about any earlier date.
A missing file proves that the resource was absent at that address at that moment, not that the operator never reserved rights by some other means.
We record public, machine-readable statements by organisations. We observe no private data, no user behaviour, and no individuals.

We are the flight recorder, not the investigator. What the record means is for others to argue.

The example above is constructed for illustration. A worked extract using a real domain, verifiable against the published root log, will be added once that domain has accumulated a meaningful observation history.