Our CI caught our own scorer writing nondeterministic rows. Here's the post-mortem.

We publish a live scorecard of our power-price forecasts against a naive baseline, every day, for every zone we serve. That page is only worth something if the numbers on it are computed correctly — and "computed correctly" is not a state you reach once, it's a state you have to keep proving. This is the story of the day we added a proof and it failed within ten minutes.

The setup: recompute everything, trust nothing

During an external audit of our forecasting system in July 2026, one checklist item read: "cross-check a handful of accuracy rows end-to-end by recomputing pinball/MAE from the raw forecasts and prices." So we built exactly that as a permanent CI job: every day, sample 20 random rows from the public accuracy table, re-derive them from the raw forecast rows and the raw settled prices — through an independent code path — and alarm if anything drifts by more than rounding.

The first run ever executed flagged 4 of 17 rows.

The bug: a nondeterministic scoreboard

Our models forecast up to seven days ahead. The public scorecard groups horizons three to seven days into one "7-day" bucket — and five different horizons were writing to that same row, in whatever order the evaluation loop happened to iterate. Last write wins. Which horizon's score you saw depended on Python dict ordering. Not wrong on average — wrong nondeterministically, which is worse, because no single spot-check would ever catch it: recompute the row and you might get the same answer for the wrong reason.

Digging into the same code path surfaced a second, older defect: on days when a delayed pipeline re-issued the same model twice, the scorer merged both runs' rows into one blended curve — a chimera that neither run ever actually produced — and scored that. Both bugs had survived every review because nobody recomputes their own scoreboard end-to-end. That is precisely the class of bug that lives forever.

The fix: determinism as a rule, not a habit

Three changes, all boring, all load-bearing:

• One winner per key. The latest issuance of a model for a given target day and horizon is the scored one — the same run customers were served longest. Duplicates never blend.
• One horizon per bucket. The 7-day bucket is scored by the largest matured horizon in it — the most conservative reading of "7 days ahead" — deterministically.
• Rows freeze. Accuracy rows rescore nightly for eight days while late price corrections can still land, then freeze by rule. The scorer refuses to touch a frozen row. If a frozen row ever disagrees with a recompute, that's an alarm to a human, never a silent rewrite.

And the recompute CI that caught it now runs daily. The proof obligation is permanent: 20 random rows, re-derived from raw data through an independent path, every morning. The day it drifts, we know the same day — not whenever the next audit happens to look.

Why publish this

Every forecast vendor claims accuracy. Almost none show you the scoring pipeline, and exactly none tell you when it was wrong. We think the trust math runs the other way: a scorecard whose failure modes are documented, monitored and published is worth more than a perfect-looking one you have to take on faith. The accuracy guarantee we sell to trading desks is only as good as the machinery in this post — so the machinery is public too.

The live scorecard is at voltcast.com/accuracy. The methodology, including the baseline definition and the shadow-model rules, is documented on the page itself. Questions or holes you think you can poke: we want to hear them.