Reference
What we store about a domain
Written for lawyers and data-protection officers, not engineers. Every technical term is explained where it first appears.
This page answers one question in full: if MachineWitness observes a domain, what exactly ends up in the archive? Not a summary, the actual fields. We walk through one worked example, beispielfirma.example, from the moment our observer connects to the point where a court could verify the record years later.
Two things are worth saying before the detail. First, we fetch only what any browser can fetch without logging in: a handful of small policy files that a site publishes for machines to read. We do not crawl page content, follow links into applications, or record anything about the people who visit a site. Second, none of this is secret. The value of this archive is not what it holds, which anyone could fetch themselves today; it is that we held it on a specific past date and can prove the record has not been altered since.
1. What we fetch, per domain, per day
Five requests. Nothing else, ever, for a domain in our broad ring.
| Resource | What it is, and why it matters legally |
|---|---|
| /robots.txt | The oldest machine-readable instruction file on the web. It tells named crawlers which parts of a site they may fetch. Since AI companies publish crawler names (GPTBot, ClaudeBot, Google-Extended and others), this file is where most sites express, or fail to express, a refusal against AI training. |
| /ai.txt | A newer, AI-specific convention for the same purpose. Not yet standardised, which is precisely why its presence or absence on a given date can be contested later. |
| /.well-known/ |
The formal machine-readable reservation of text-and-data-mining rights (TDM Reservation Protocol, W3C). Under Art. 4(3) of the EU DSM Directive a rights-holder's reservation is effective only if machine-readable; the Hamburg Higher Regional Court confirmed in December 2025 that natural-language terms alone do not suffice. This file is the cleanest way to make that reservation. |
| /llms.txt | An emerging convention addressed at large language models directly. |
| / (homepage) | Fetched for its response headers and the first few kilobytes only, never the full page. Some sites express their TDM reservation in an HTTP header or a meta tag rather than a file. |
Each response is capped at a hard byte limit. These files are normally a few kilobytes; the cap exists so that a misconfigured server cannot make us download something large.
2. What one observation contains
An observation is one recording of one resource at one moment. Here is a complete one for our example domain, in the form the archive holds it.
The content itself
GET https://beispielfirma.example/robots.txt → HTTP 200, 412 bytes User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
The exact bytes the server delivered, stored unchanged and compressed. Not our summary of them, not a rendering: the bytes. In a dispute the question is never "what does the file say" but "what did it say on 14 August 2027".
The fingerprint
payload_sha256 = 9f2a41d7c8e05b3a…7d1e6084bb93c2f5 payload_size = 412 bytes
A SHA-256 hash is a short fixed-length code computed from the exact bytes of a file. Change a single character and the code changes completely, and it is computationally infeasible to construct a different file with the same code. It functions as a fingerprint of one specific version of a document. This is also the filing number: the content is stored under its own fingerprint, so nothing can be substituted without the label ceasing to match.
The circumstances of delivery
observed_at = 2027-08-14 03:00:11 UTC
observer_id = witness-1
status_code = 200
final_url = https://www.beispielfirma.example/robots.txt (after redirect)
http_version = HTTP/1.1
headers = content-type: text/plain; charset=UTF-8
cache-control: max-age=600
x-served-by: cache-fra-…
(complete response headers, verbatim)
tls_version = TLSv1.3
peer_cert = subject: *.beispielfirma.example
issuer: GlobalSign Atlas R3 DV TLS CA
valid: 2027-06-01 → 2028-06-01
sha256: 3b8c07f2ad91…e5240ac71f6b88
observer_addr = the IP address our observer connected from
peer_addr = the IP address that answered
This is what lawyers call chain of custody: not merely "this file existed" but "this server, presenting this TLS certificate, delivered these bytes to this observer at this moment". The TLS certificate matters because it ties the delivery to a party who could obtain a certificate for that domain. All of it is sealed together with the content, so no part can be swapped later.
The change flag
changed = true (fingerprint differs from the previous observation)
Set when today's fingerprint differs from yesterday's for the same resource. It is the reason this archive exists: these files change silently, without a public history, and the change itself is often the disputed fact.
3. What an absent file looks like
A record of "nothing was there" is evidence too, and it is recorded with the same care as content. If our example domain serves no ai.txt:
GET https://beispielfirma.example/ai.txt → HTTP 404, no payload stored headers and TLS chain recorded as above
This distinction carries real weight. "The site had published no machine-readable AI reservation on that date" is frequently the decisive fact in a TDM dispute, and it can only be shown by someone who looked and wrote down that they found nothing. Two further cases are recorded just as explicitly: a server that refuses our observer (HTTP 403) produces a record of the refusal, not of content; and a resource that a site's own robots.txt tells us not to fetch is recorded as deliberately not fetched. We honour that instruction and log that we did.
4. How the record is sealed
An archive kept by an interested party proves little on its own: we could, in principle, rewrite our own database. Four steps address that objection, each independent of the others.
Every observation of the day is bound into one figure
All fingerprints recorded that day are combined pairwise, level upon level, until a single daily root remains that depends on every record beneath it. Change any one observation afterwards and the root no longer matches. This is the construction used by the public logs that browsers rely on to audit TLS certificates (RFC 6962).
That figure is deposited outside our control
The daily root is timestamped into the Bitcoin blockchain via OpenTimestamps and, separately, by an independent RFC 3161 time-stamping authority. We cannot rewrite either. This is what upgrades the seal from an internal claim to external proof: it shows the record existed no later than that day.
The root is published
Each daily root appears in our public root log, with stable URLs and machine-readable form. Anyone may keep their own copy the day it appears; several third-party systems capture it automatically.
A second, independent witness observes the same targets
A separate system, on different infrastructure with a different operator key, records independently and publishes its own daily root. Matching testimony from two witnesses who never shared a database is qualitatively stronger than one witness repeating himself. This is under construction and not yet in operation; we say so here rather than let the architecture description imply otherwise.
5. What an evidence extract contains
The archive is not browsable, by design. What we produce on request, for a specific domain and date range, is an evidence extract: a package containing
- the stored payload for each observation in the range, byte for byte;
- the complete sealed circumstances of delivery, as set out in section 2;
- the fingerprint of each observation;
- an inclusion proof: a short mathematical receipt showing that this specific observation is contained in that day's root, checkable without trusting us and without access to the rest of the archive;
- the daily root and the external timestamp receipts;
- step-by-step verification instructions that any competent technical expert can follow with standard tools.
The point of the last item is worth stating plainly: the extract is designed so that the opposing party's expert can verify it. A record only one side can check is not evidence.
6. If someone asks us to erase
Most of what we hold contains no personal data: these are technical policy files published by organisations. Where a legitimate erasure request under Art. 17 GDPR does concern archived content, we perform what we call a tombstone: the stored content is deleted and replaced by a marker. Deleted means deleted, and no backup restores it.
What remains is the fingerprint, the seal, and the proofs. The consequence is worth understanding precisely: afterwards, nobody can learn what the file said, but it remains provable that a file with exactly this fingerprint stood at this address on this date, and that the surrounding records of that day are untouched. Retaining the fingerprint rests on Art. 17(3)(e) GDPR, because deleting it would break the proof chain for every other observation sealed that day, including those belonging to unrelated third parties.
On request we also exclude a domain from all future observation, with or without an erasure. Both are free, informal, and need no justification. See Privacy and Crawler & contact.
7. What this record does not prove
Stating the limits precisely is part of being a credible witness.
- It shows what a server delivered to our observer, at the recorded moments. It says nothing about the intervals between observations, and nothing about what other visitors saw.
- It shows content and circumstances of delivery, not authorship, intent, or lawfulness. Whether a reservation was effective, whether a crawler respected it, and what follows legally are questions for counsel and the court.
- External anchoring proves a record existed no later than its anchor date. It proves nothing about any earlier date.
- A missing file proves that the resource was absent at that address at that moment, not that the operator never reserved rights by some other means.
- We record public, machine-readable statements by organisations. We observe no private data, no user behaviour, and no individuals.
We are the flight recorder, not the investigator. What the record means is for others to argue.