Blind-300.

01 · WHAT IT IS

Three hundred news articles, each read by two human annotators working independently against the same written rubric, then adjudicated into a single gold answer: which articles describe a real-world event, and for each event, who did what to whom, when, and where.

It exists because we needed to know whether our own extraction was any good, and there was nothing to measure against that we had not built ourselves. A benchmark you grade yourself on is not a benchmark.

02 · WHAT SHIPS, AND WHAT DELIBERATELY DOESN'T

Annotations only. No article text, anywhere.

Each record carries the article's URL, title, publish date, and our labels. If you want the text, you fetch it from the URL yourself, exactly as we did.

That constraint is the whole design. Distributing article bodies would mean redistributing other people's copyrighted work, so we don't — publishers keep every right they had before we read anything. It also means the benchmark can be released at all, rather than living forever as an internal certification set.

The cost is honest to state: link rot will erode it. Some fraction of those URLs will not resolve in five years, and the set will decay. We judged a decaying public benchmark better than a permanent private one.

03 · THE SPLIT, AND WHY IT'S COMPUTED RATHER THAN CHOSEN

The assignment is content-addressed: an article is in the development half if and only if the first byte of sha256(article_id) is less than 128. Nothing was seeded, sampled, or hand-picked, which means we had no freedom to sort the easy cases into the half we publish. Anyone with the public id list can recompute the entire split and check that.

The development half — 159 records, 53 in scope, 102 events — is yours. Develop against it, tune against it, overfit it if you like; that's what it's for.

04 · THE COMMITMENT

The withheld half is not published. What is published is the SHA-256 of the file:

54c04dac473ea13b417191a8bfb08c121f01f071e3591a9f84bdab39a5c6245c

That hash is the point of the whole exercise. Any score we later report against the withheld set can be checked against a file we were publicly committed to before the run — so we cannot quietly swap in a friendlier test set, drop the items we did badly on, or relabel after seeing results. The commitment file also lists the 141 withheld ids, so you can verify that the withheld half is the exact complement of the published half and that nothing was held back beyond it.

The exporter enforces this rather than trusting us to: it hard-errors, naming both hashes, rather than overwrite an existing withheld file whose bytes differ. A published commitment cannot be rewritten by accident, and rewriting it deliberately requires deleting the guard.

If you want your own system scored against the withheld half, ask. The file is held privately by the authors; its location is not published.

05 · AGREEMENT BETWEEN THE ANNOTATORS

A benchmark whose own labels are a coin flip is worth nothing, so this was measured before the set was promised as a contribution — against a bar written down in advance.

The pre-registered bar was article agreement ≥ 0.6 and audit uphold ≥ 0.8. Both cleared.

Two things the 0.552 does not mean

It is not comparable to the 0.807, and it is not "the annotators agreed on 55% of the data."

It is a rate over the in-scope minority. Only 95 of the 300 articles are in scope in final gold — two-thirds of news articles describe no event at all. Of the 242 articles where both passes agreed, 197 are cases where both independently said "nothing here," and an agreed-empty article contributes nothing to the pair numerator. So 0.552 measures agreement on the small, hard subset where something actually happened.

The denominator mixes units. A presence dispute — one annotator sees an event the other doesn't — is single-sided and isn't an aligned pair, yet it sits in the denominator alongside true pairs, which deflates the rate. Counting those as disagreement gives 0.552; dropping them gives 0.669. Both are computable from the artifacts; we publish the pessimistic one as the headline and the bracket alongside it.

The disputes break down as 32 presence, 4 type, and 46 field.

And one number we cannot give you

These are raw percent agreement, not chance-corrected. There is no Cohen's kappa, and there never will be: computing it needs the raw per-pass label files, which were gitignored and no longer exist on any machine we have. That is a permanent gap in the record caused by an ordinary mistake, and we would rather say so than publish an estimate dressed as a measurement.

06 · HOW WE KNOW THE RED LINE HELD

The rule was: no article body text anywhere in the released files. It is enforced three ways — records built field-by-field from a fixed six-field allowlist rather than by copying and deleting; a test asserting that no 120-character window from any of the 300 article bodies appears anywhere in the released directory; and a scan of every released file on disk, including the README and the licence.

That last scan carries a negative control: it asserts it still catches a planted body fragment quoted in prose inside a synthetic file. Without the control, "zero leaks found" and "the scan matches nothing at all" are indistinguishable, and the test would have been reassuring and empty.

We know that because it happened. The first version of the release README contained a grep proof that quoted a fragment of an article body — putting article text into the very directory that claims to have none, in exactly the shape the scan was supposed to catch. The test at the time re-derived records in memory and was structurally incapable of seeing it. Both are fixed; the scan now reads what was actually written to disk.

A similar near-miss is worth recording: our first implementation counted an upheld spot-check by the wrong token, and would have reported 0/15 instead of 15/15 — demoting the benchmark on a false negative. It was caught because the number was implausible enough to check.

07 · LICENCE

The annotations are our own original work against our own rubric, released under CC-BY 4.0. URLs, titles and publication dates are facts about publications. The articles themselves are not ours and are not distributed; their publishers retain all rights.

08 · KNOWN LIMITS