01 · THE ASSUMPTION
Any system that reasons from historical news has to actually retrieve historical news, and web pages do not stay put. We had modelled that decay at 6% a year — a placeholder, marked provisional, carried through a cost model that several decisions depended on.
Provisional numbers have a way of becoming load-bearing quietly, so we went and measured it.
02 · WHAT WE MEASURED FIRST
A probe of 240 articles spanning 2016–2026, fetched through the same production chain the real pipeline uses. Availability by age:
AVAILABILITY AT THE ORIGINAL PUBLISHER · BY AGE
That fits about 8.3% a year — meaningfully worse than the 6% we'd assumed, and worse in the direction that hurts, since historical depth is exactly what a base rate is made of. Note the floor: decay stalls around 35% at eight years. What survives eight years tends to keep surviving.
We had a better number than we started with. We also had a mistake.
03 · THE MISTAKE
That measures the origin, not availability. It answers "is this article still where it was published?" — which is a fact about publishers' CMS migrations, not about whether we can read the article.
It is a precise measurement of the wrong quantity, and it would have been easy to keep. It was internally consistent, it moved a number we knew was provisional, and it confirmed a pessimism we already held. Nothing about it felt wrong.
04 · WHAT HAPPENS WHEN YOU ASK THE RIGHT QUESTION
Adding the Internet Archive, queried through its CDX API, lifts pooled availability to 89% — and does something stranger than lift it. It flattens the age curve, then tips it over:
ORIGIN + ARCHIVE · BY AGE
Read that twice. Ten-year-old news is more reliably retrievable than last week's. The archive has had a decade to crawl the old material and almost no time for the new, so the scarcity everyone assumes — old is hard, recent is easy — runs backwards once an archive is in the loop.
The decay term we were about to bake in, availability falling exponentially with age, is not the shape of the problem at all. Beyond roughly two years it is approximately flat.
05 · THE SECOND NAIVETY
"Use the archive, then" is its own trap, because availability is not throughput.
Measured over two six-hour probes, the archive sustains roughly 350–900 articles a day for a single identified, polite client. A corpus is retrievable in principle and still years away in practice. If you are planning against archive coverage, the number that sets your calendar is the drip rate, not the coverage percentage — and the polite thing and the fast thing point in opposite directions.
06 · WHAT COMMON CRAWL ADDS
Measured against origin plus archive, Common Crawl holds ~13% of the articles we'd extract and supplied exactly four of 240 we couldn't already fetch. That verdict was right and nearly useless, because it answers "what does this add to a baseline that already contains everything?" — and the answer to that is "not much," for almost any source.
Measured incrementally, CC-NEWS carries 6.0–8.0pp of unique additions over origin, and CC-MAIN a further 5.1–7.7pp.
07 · WHAT THIS MEANS IF YOU BUILD ON NEWS URLS
Don't model decay as exponential-with-age.
With an archive it is roughly flat past two years. Without one it is steep, and the steepness is the reason to have one.
Your newest data is your most fragile.
Counterintuitive, and it argues for fetching and keeping aggressively at publication time. Retrieval is a one-time cost if you retain the bytes; it recurs forever if you don't.
Coverage and throughput are different questions.
Answer both before you commit to a calendar.
Check your baseline before you reject a source.
Against a baseline that already contains everything, almost any source measures as adding nothing. Ours measured four of 240.
It also sets a real limit on URL-keyed datasets, including our own. Blind-300 is keyed by URL and distributes no article text, so it decays exactly like this — which is why it asks anyone reporting a score to report how many of the articles they could fetch alongside it. A benchmark that quietly loses 10% of its items between two evaluations is not comparing what it appears to compare.
08 · METHOD AND LIMITS
Availability probe: n=240 prefilter-passed news articles, 2016–2026, fetched through the production chain rather than an idealised client. Archive throughput: n=610 events across two six-hour probes, one identified client. Common Crawl coverage: three full calendar days for CC-NEWS, with CC-MAIN measured incrementally against origin plus CC-NEWS.
The honest caveats: 240 is a small sample and the confidence intervals on any single age bucket are wide. This is mainstream news that passed a relevance filter, not a random sample of the web, and the web at large almost certainly rots faster. Everything here measures our retrieval path — a different client, with different politeness limits and different failure handling, will get different numbers. And the archive is one nonprofit, not a guarantee; a strategy that assumes it will always be there and always be free is making an unpriced bet.
Measurements: availability probe 2026-07-30; horizon, throughput and coverage figures 2026-08-07. The 6%/yr assumption they replaced was published as provisional and is superseded.