PYTHIOS LABS
Experiments, ablations, and negative results from building the engine — and the vocabulary to read them with.
What we built, what we measured, and what we're releasing. Negative results and near-misses included — a scoreboard that only shows wins isn't one.
1.0Findings →
The vocabulary of calibrated forecasting, defined plainly. If a term appears in one of our findings and isn't here, tell us — hello@pythios.ai.
2.0Glossary
2.1FORECASTING
How often something happens across the reference class, before you know anything specific about the case in front of you. If EU sanctions packages have historically followed a given trigger 4 times in 10, the base rate is 0.4. Most bad forecasting is a base rate ignored in favour of a vivid story.
The set of past situations judged similar enough that their outcomes tell you something about this one. Choosing it is the hard part and it is a judgement, not a calculation: too narrow and you have three examples, too broad and you are averaging over situations that share nothing but a label. A stated reference class is what separates a forecast from a guess.
The written rule, fixed in advance, that decides whether a question resolved yes or no. "A new or expanded sanctions measure by the European Union naming Belarus, occurring on or before 2026-09-01" is a resolution criterion. "Will things get worse?" is not. A probability attached to an unresolvable question cannot be scored, and anything that cannot be scored cannot be wrong.
The window a forecast covers. Horizons are not interchangeable — the same question at 30 days and 12 months is two different questions with two different base rates, and pooling them produces a number that describes neither.
How far forecasts move away from the base rate. This is the counterweight to calibration: a forecaster who answers with the base rate every single time is perfectly calibrated and completely useless. Skill requires being calibrated and sharp — moving away from the base rate, and being right when you do.
2.2EVALUATION
The property that your stated probabilities mean what they say: across everything you called 70%, about 70% should have happened. Calibration is a measure of honesty rather than intelligence — it asks whether the number can be trusted at face value, not whether it was clever.
Forecasts grouped by the probability given — everything in 0.6–0.7, everything in 0.7–0.8 — so that predicted frequency can be compared against observed frequency within each band. Plotted, these make a reliability diagram: the diagonal is perfect calibration, above it is underconfidence, below it is overconfidence.
The mean squared error of probabilistic forecasts. 0 is perfect; 0.25 is what you get by saying "50%" to everything; 1.0 is being confidently wrong every single time. Lower is better. It rewards calibration and sharpness together, which is why it is the number we report.
A scoring rule you cannot game by lying. Under a proper rule — Brier is one — your expected score is best when you report the probability you actually believe, so there is no strategic advantage in shading a number toward safety. Improper rules quietly reward hedging, which is how scoreboards end up full of forecasts clustered near 50%.
Two ways of being wrong. Recall is the share of the things you should have caught that you did catch. Precision is the share of the things you flagged that were real. They trade against each other, and which one you protect is a values decision: in a filter that feeds a forecasting engine, a missed relevant article is invisible and permanent, while a false positive merely costs compute. We run our relevance classifier at recall 1.000 for exactly that reason.
The cut-off at which a score becomes a yes/no decision — the point on the curve where you choose your precision/recall trade. Publishing a model's accuracy without its threshold is publishing half a result.
A set of examples labelled by hand, carefully, and then frozen — the ruler you measure the system against. Its value depends entirely on not being tuned against: the moment you adjust the gold set because the model disagreed with it, you are grading the exam with the answers the student wrote.
Testing on a period the model never saw during training, rather than on a random sample of the same period. Random splits leak: news from the same week shares wording, framing and events, and a model can score well by recognising a story rather than by understanding it. A temporal holdout is the honest version of the test.
How often two humans labelling the same material independently agree. It bounds everything downstream — a model cannot be meaningfully measured as more accurate than the humans who wrote its answer key are consistent. Reported agreement should always state whether it is raw percent agreement or chance-corrected, because the two differ substantially and only one of them is impressive.
A test asking "is the new thing not meaningfully worse than the old thing", within a margin declared before the test is run. This is the correct question when replacing something expensive with something cheap: you do not need the cheap version to win, only to not lose by more than you can afford. Declaring the margin afterwards turns it into a rationalisation.
2.3PIPELINE
Reading an article with a language model and writing down what happened in structured form: actor, action, target, time, place. This is the step that turns prose into something you can count, and it is where most of the error in a news-derived system originates.
The substrate's three nouns. An actor is an entity that does things — a state, an organisation, a person. An event is something that happened, with participants and a time. A claim is an assertion that an event occurred, attributed to the source that made it. The distinction between event and claim is load-bearing: reporting is evidence that something happened, not the happening itself, and a system that collapses the two cannot represent a source being wrong.
The chain from a published number back to the specific source that produced it. Provenance is what makes a forecast auditable rather than merely confident — without it, a probability is an opinion with a decimal point.
The steady disappearance of published material from the web. Measured across a decade of news, availability at the original publisher falls from about 83% at publication to roughly 35% at ten years — around 8.3% a year. It is the reason historical depth is a genuine asset rather than something anyone can assemble on demand.