Export
Download a released day, or any filtered slice of the corpus. The file is streamed, so a
whole day is a download rather than a wait, and every file carries the manifest of every
day it contains — the counts and the
snapshot_content_hash — so a downloaded
corpus traces back to the snapshot it came from.
What is in the file
| Field | Meaning |
|---|---|
| record | `manifest` for a day's header row, `content` for a corpus item. |
| day | The UTC day the item belongs to, and the snapshot it was frozen into. |
| kind | `test` or `production`. |
| id | The platform's stable identifier for the item. |
| author_hotkey | The contributing miner. Empty on production rows: production content has no author. |
| created_at | Unix seconds, UTC. |
| content | The item as stored. |
| content_encoding | `verbatim`, or `csv-formula-guarded` when a CSV cell was prefixed to stop a spreadsheet executing it. |
| snapshot_content_hash | The day's content hash, repeated on every row so a row traces back to its snapshot. |
Two formats, and they are not interchangeable
One JSON object per line, content byte-exact. This is the form to use if you intend to
recompute a day's snapshot_content_hash:
nothing in it is altered.
A fixed rectangle for spreadsheets. A cell that begins with
=,
+,
- or
@ would be executed as a formula by a
spreadsheet, and this corpus contains text written to attack whatever opens it — so
those cells are prefixed with a quote and the row's
content_encoding column says
csv-formula-guarded instead of
verbatim. A guarded cell is not
byte-identical to the stored item, which is why the CSV is not the form to hash.
An export only ever contains released content. Asking for a day inside the
2-day embargo returns that day's manifest with
test_content_released set false and no test
rows — a short file that says why it is short, rather than an empty one that looks broken.
Production content is unattributed. The
author_hotkey column is empty on every
production row because production content has no author of any kind, not because one is
missing: it is the negative half of the corpus, and attributing it would hand every
validator a lever on scoring.