How we score forecasts without fooling ourselves

Forecast accuracy claims are the easiest numbers in this industry to fake — usually not by lying, but by fooling yourself. This paper documents the five rules we run our own scoring under, because a claim you can't audit is marketing, not measurement.

1 · Walk-forward or it didn't happen

Every backtest row on our board is strictly walk-forward: the model is trained only on data from before each simulated issue day, steps forward day by day, and is retrained on schedule exactly as production would have been. A model that sees one minute past its issue day is disqualified. Backtest rows are labeled backtest forever — they never blend into the live record.

2 · Features must be knowable at issue time

The subtle leaks live in the features. Weather inputs come from a previous-runs archive — the forecast as it existed the day before delivery, not today's improved reanalysis. Grid drivers are lagged to the last complete day at issue time (a same-day average of realized flows would leak the half-day that hadn't happened yet at the 13:00 auction). Revision-aware sources are pinned to the revision that was public at issue. Each feature's knowability rule is documented per column in the Quant feature matrices — the same rows our own models train on.

3 · The baseline must be embarrassing to lose to

We score against naive persistence: replay the issue day's realized curve for the target day. It costs nothing, anyone can build it, and on price-regime days it is brutally hard to beat (see our July 12 post-mortem). A vendor who picks a weaker baseline — or none — is telling you something.

4 · Burn-in is labeled, losses stay up

A live record under three days is statistically meaningless in either direction, so it's labeled burn-in and excluded from headline numbers. Retired model versions keep their record on the board, labeled. Losing days, losing months, losing generations — all stay public. The board is append-only in spirit: we add context, we never remove results.

5 · A model serves only when it wins

New generations run in shadow and flip to serving only on walk-forward evidence against the incumbent — per zone, not globally. The same rule holds across the family: our temperature model (volt-temp-1) serves only while it beats the raw 51-member ECMWF ensemble it post-processes; our renewables model applies a validated shrinkage weight that collapses to the TSO's own forecast wherever we don't add value. Independent verification exists on top: daily submissions to Energy-Arena (externally timestamped, third-party ground truth) and the open Forecast Bench, whose scoring code is published verbatim.

The commercial consequence keeps us honest: if our forecast doesn't beat the baseline in your zone for a month, that zone is credited automatically. The scorecard isn't a marketing page that could drift from reality — it is the billing trigger.