Reference

What we store about a domain

Written for lawyers and data-protection officers. Every technical term is explained where it first appears.

This page answers one question in full: if MachineWitness observes a domain, what exactly ends up in the archive? Not a summary, the actual fields. We walk through one worked example, beispielfirma.example, from the moment our observer connects to the point where a court could verify the record years later.

Before any of the files below are fetched, we read the domain's robots.txt and obey it. Where it tells us not to fetch a file, we do not fetch it, and the record says so: the observation for that day is recorded as disallowed_by_robots, with the provenance policy: robots_respected, in place of a response. That is a different statement from a server that failed to answer, and the archive keeps the two apart. On 10 August 2026 this applied to roughly 14,000 of the 128,347 domains in the ring, for the three files other than robots.txt itself. We could fetch them anyway — the files are public — and we do not.

Two things are worth saying before the detail. First, we fetch only what any browser can fetch without logging in: a handful of small policy files that a site publishes for machines to read. We do not crawl page content, follow links into applications, or record anything about the people who visit a site. Second, none of this is secret. The value of this archive is not what it holds, which anyone could fetch themselves today; it is that we held it on a specific past date and can prove the record has not been altered since.

1. What we fetch, per domain

Five requests. Nothing else, ever, for a domain in our broad ring.

Resource	What it is, and why it matters legally
/robots.txt	The oldest machine-readable instruction file on the web. It tells named crawlers which parts of a site they may fetch. Since AI companies publish crawler names (GPTBot, ClaudeBot, Google-Extended and others), this file is where most sites express, or fail to express, a refusal against AI training.
/ai.txt	A newer, AI-specific convention for the same purpose. Not yet standardised, which is precisely why its presence or absence on a given date can be contested later.
/.well-known/tdmrep.json	The formal machine-readable reservation of text-and-data-mining rights (TDM Reservation Protocol, W3C). Under Art. 4(3) of the EU DSM Directive a rights-holder's reservation is effective only if machine-readable; the Hamburg Higher Regional Court confirmed in December 2025 that natural-language terms alone do not suffice. This file is the cleanest way to make that reservation.
/llms.txt	An emerging convention addressed at large language models directly.
/ (homepage)	Fetched for its response headers and the first few kilobytes only, never the full page. Some sites express their TDM reservation in an HTTP header or a meta tag rather than a file.

Each response is capped at a hard byte limit. These files are normally a few kilobytes; the cap exists so that a misconfigured server cannot make us download something large.

Method change, effective 3 August 2026

From that day forward the archive applies the rules below. They are stated here rather than left implicit, because a change in method changes what a record from a given date can be used to prove. Nothing before 3 August 2026 was altered; earlier days keep the method that was in force when they were sealed.

Two rings. The rules differ between a small core ring and a large broad ring. The broad ring is defined by a stated criterion, not by selection: all EU domains within the Tranco top-1M list, as retrieved on 2 August 2026. The core ring is a much smaller set of domains observed more closely. It is assembled by hand and we do not yet publish criteria for it, so it should be read as a working selection rather than a curated register.

The four opt-out files are still fetched daily for every domain. That is the part that carries legal weight, and it is unchanged.
The homepage is fetched weekly, not daily, for domains in the broad ring. The honest consequence: where a site expresses its reservation only in a homepage meta tag or HTTP header, our broad-ring record of that reservation is precise to the week, not to the day. For the core ring nothing changes — there the homepage is still fetched daily.
Byte caps differ by ring. Core ring: uncapped. Broad ring: 256 KB, with anything beyond that marked truncated. What is kept is always the beginning of the response, where these signals live.
Error responses are recorded without their body. Status code, all headers, size and content hash are kept, so the fact and shape of the error remain provable; the error page itself is not stored. One exception: 403 Forbidden bodies are kept up to 32 KB, because a refusal aimed at crawlers is sometimes stated in that page.

2. What one observation contains

An observation is one recording of one resource at one moment. Here is a complete one for our example domain, in the form the archive holds it.

The content itself

GET https://beispielfirma.example/robots.txt
→ HTTP 200, 412 bytes

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The exact bytes the server delivered, stored unchanged and compressed. Not our summary of them, not a rendering: the bytes. In a dispute the question is never "what does the file say" but "what did it say on 14 August 2027".

The fingerprint

payload_sha256 = 9f2a41d7c8e05b3a…7d1e6084bb93c2f5
payload_size   = 412 bytes

A SHA-256 hash is a short fixed-length code computed from the exact bytes of a file. Change a single character and the code changes completely, and it is computationally infeasible to construct a different file with the same code. It functions as a fingerprint of one specific version of a document. This is also the filing number: the content is stored under its own fingerprint, so nothing can be substituted without the label ceasing to match.

The circumstances of delivery

observed_at   = 2027-08-14 03:00:11 UTC
observer_id   = witness-1
status_code   = 200
final_url     = https://www.beispielfirma.example/robots.txt   (after redirect)
http_version  = HTTP/1.1
headers       = content-type: text/plain; charset=UTF-8
                cache-control: max-age=600
                x-served-by: cache-fra-…
                (complete response headers, verbatim)
tls_version   = TLSv1.3
peer_cert     = subject:  *.beispielfirma.example
                issuer:   GlobalSign Atlas R3 DV TLS CA
                valid:    2027-06-01 → 2028-06-01
                sha256:   3b8c07f2ad91…e5240ac71f6b88
                (fingerprint of the server certificate,
                 plus the five parsed fields above)
observer_addr = the IP address our observer connected from
peer_addr     = the IP address that answered

This is what lawyers call chain of custody: not merely "this file existed" but "this server, presenting this TLS certificate, delivered these bytes to this observer at this moment". The TLS certificate matters because it ties the delivery to a party who could obtain a certificate for that domain. All of it is sealed together with the content, so no part can be swapped later.

Precisely what is recorded here, and what is not. We record the SHA-256 fingerprint of the server certificate together with the five parsed fields shown above. We do not store the certificate itself, and we do not store the intermediate certificates that make up the chain to a root authority. A fingerprint alone cannot be re-verified without the certificate it was taken from; where the certificate is needed, it can generally be retrieved independently from the public Certificate Transparency logs (RFC 6962), which is the same audit infrastructure browsers rely on. We state this rather than let the shorter phrase "TLS chain" stand, because a record is worth only what it actually contains. Recording the full chain is under evaluation; if it is added, this passage will be updated with the date from which it applies.

The change flag

changed = true   (fingerprint differs from the previous observation)

Set when today's fingerprint differs from yesterday's for the same resource. It is the reason this archive exists: these files change silently, without a public history, and the change itself is often the disputed fact.

3. What an absent file looks like

A record of "nothing was there" is evidence too, and it is recorded with the same care as content. If our example domain serves no ai.txt:

GET https://beispielfirma.example/ai.txt
→ HTTP 404, no payload stored
   status, complete headers, payload fingerprint and
   TLS details recorded as above

This distinction carries real weight. "The site had published no machine-readable AI reservation on that date" is frequently the decisive fact in a TDM dispute, and it can only be shown by someone who looked and wrote down that they found nothing. Two further cases are recorded just as explicitly: a server that refuses our observer (HTTP 403) produces a record of the refusal, not of content; and a resource that a site's own robots.txt tells us not to fetch is recorded as deliberately not fetched. We honour that instruction and log that we did.

4. How the record is sealed

An archive kept by an interested party proves little on its own: we could, in principle, rewrite our own database. Four steps address that objection, each independent of the others.

Step 1 · same day

Every observation of the day is bound into one figure

All fingerprints recorded that day are combined pairwise, level upon level, until a single daily root remains that depends on every record beneath it. Change any one observation afterwards and the root no longer matches. This is the construction used by the public logs that browsers rely on to audit TLS certificates (RFC 6962).

Step 2 · submitted same day

That figure is deposited outside our control

The daily root is submitted the same day to OpenTimestamps, which aggregates it towards the Bitcoin blockchain, and separately to an independent RFC 3161 time-stamping authority, which returns a signed token immediately. Since 31 July 2026 a qualified eIDAS time-stamp is obtained as well. We cannot rewrite any of them. This is what upgrades the seal from an internal claim to external proof: it shows the record existed no later than that day.

Status of the Bitcoin anchor: completed on 7 August 2026. The two time-stamping authorities return their proof at once, and those tokens are held here. OpenTimestamps works in two stages: the root is submitted immediately, but the resulting receipt only becomes self-contained once the Bitcoin confirmation is fetched back into it. Until 7 August 2026 we had not performed that second step, and an earlier version of this passage said so. It has now been carried out for every sealed day on both witnesses, and a daily job performs it from here on, so the receipts stored here reference Bitcoin block headers rather than calendar servers. The receipts were changed by this step, but what they attest was not: the hash each receipt witnesses was recorded before and after and is unchanged for every day. The Bitcoin path no longer depends on those calendar servers remaining reachable. The RFC 3161 and qualified eIDAS anchors were unaffected throughout and stand on their own.

Step 3 · same day

The root is published

Each daily root appears in our public root log, with stable URLs and machine-readable form. Anyone may keep their own copy the day it appears; several third-party systems capture it automatically.

Step 4 · operating since 2026-08-04

A second, independent witness observes the same targets

A separate system, on different infrastructure in another country and with a different operator key, records independently and publishes its own daily root. Neither machine holds credentials for the other and neither can write to the other's database. Two accounts kept independently are harder to dismiss than one witness repeating himself.

Two limits stated plainly. First, this began on 4 August 2026: every day sealed from 22 July to 3 August 2026 rests on the first witness alone, and the public log shows that gap rather than hiding it. Second, the two daily roots are never identical, by construction — each witness crawls on its own schedule, so the set of observations behind each root differs. A comparison is therefore made observation by observation, not by checking whether two roots match; identical roots would in fact indicate that the two systems were not independent.

5. What an evidence extract contains

The archive is not browsable, by design. What we produce on request, for a specific domain and date range, is an evidence extract: a package containing

the stored payload for each observation in the range, byte for byte;
the complete sealed circumstances of delivery, as set out in section 2;
the fingerprint of each observation;
an inclusion proof: a short mathematical receipt showing that this specific observation is contained in that day's root, checkable without trusting us and without access to the rest of the archive;
the daily root and the external timestamp receipts;
step-by-step verification instructions that any competent technical expert can follow with standard tools.

The point of the last item is worth stating plainly: the extract is designed so that the opposing party's expert can verify it. A record only one side can check is not evidence.

How one is requested, the terms it is issued under, and what it costs are set out on the evidence extract page. Whether this archive observes a given domain at all can be checked first, at no cost, with the coverage check.

6. If someone asks us to erase

Most of what we hold contains no personal data: these are technical policy files published by organisations. Where a legitimate erasure request under Art. 17 GDPR does concern archived content, we perform what we call a tombstone: the stored content is deleted and replaced by a marker. Deleted means deleted, and no backup restores it.

What remains is the fingerprint, the seal, and the proofs. The consequence is worth understanding precisely: afterwards, nobody can learn what the file said, but it remains provable that a file with exactly this fingerprint stood at this address on this date, and that the surrounding records of that day are untouched. Retaining the fingerprint rests on Art. 17(3)(e) GDPR, because deleting it would break the proof chain for every other observation sealed that day, including those belonging to unrelated third parties.

On request we also exclude a domain from all future observation, with or without an erasure. Both are free, informal, and need no justification. See Privacy and Crawler & contact.

7. What this record does not prove

Stating the limits precisely is part of being a credible witness.

It shows what a server delivered to our observer, at the recorded moments. It says nothing about the intervals between observations, and nothing about what other visitors saw.
It shows content and circumstances of delivery, not authorship, intent, or lawfulness. Whether a reservation was effective, whether a crawler respected it, and what follows legally are questions for counsel and the court.
External anchoring proves a record existed no later than its anchor date. It proves nothing about any earlier date.
A missing file proves that the resource was absent at that address at that moment, not that the operator never reserved rights by some other means.
We record public, machine-readable statements by organisations. We observe no private data, no user behaviour, and no individuals.

We are the flight recorder, not the investigator. What the record means is for others to argue.

The example above is constructed for illustration. A worked extract using a real domain, verifiable against the published root log, will be added once that domain has accumulated a meaningful observation history.