A backtest should recreate what could have been known at each forecast date. Freeze data availability, model rules and forecast horizons, then compare predictions with an explicitly chosen outcome version. Using later corrections or future-derived inputs can make historical performance look unrealistically strong.
Reconstruct the information available at the forecast date
An event date and the date its record became available are different. A delivery completed on Monday may not enter the system until Thursday. A Tuesday forecast cannot legitimately use Thursday's update just because its event date is earlier. Record availability timestamps, source versions and reporting lags. If historical versions do not exist, describe the limitation rather than claiming a fully faithful reconstruction.
Freeze the model specification and the process for selecting its settings. A backtest can leak future information through feature construction, scaling, category definitions or repeated model tuning, even when the target column is hidden. Check that rolling averages only use eligible past observations. For each forecast origin, retain the exact input window, parameters and prediction so a reviewer can reproduce what the method actually did.
Compare predictions at the same horizon and scope
A forecast one day ahead is not comparable to one made a month ahead without explaining the different information and difficulty. Define the horizon, unit, population and aggregation level. Report performance by horizon when operations use several. Reconcile additions or removals of locations so apparent accuracy does not arise from excluding the hardest cases after their outcomes become known.
Choose a transparent benchmark, such as the most recent comparable period, and evaluate it with the same information limits. Define error measures and their units. Absolute error treats overprediction and underprediction symmetrically; the operational consequences may not be symmetric. Percentage errors can be problematic near zero. Use measures that fit the decision and report systematic overprediction or underprediction alongside an overall error summary.
Keep evaluation separate from retrospective improvement
Decide which outcome version represents the evaluation target. An early operational estimate may be the value a planner actually sees, while a later corrected total may better represent the final outcome. Both comparisons can be useful if labelled. Do not silently replace earlier inputs with corrected data and present the resulting prediction as the forecast that would have been available at the time.
Reserve a genuinely later period for evaluating choices made during development where feasible. Repeatedly selecting the method that wins on the same historical window can overfit that window. Document exceptions, failed forecasts and missing outcomes rather than dropping them for convenience. The conclusion should state observed historical performance under the specified reconstruction, with conditions that could make future performance different, rather than promising that past error will continue unchanged.
A fictional weekly demand backtest
A fictional scenario to explain the method, not a customer result or a market benchmark.
A fictional maintenance planner forecasts 110 jobs on 1 June for the following week. The latest available demand history ends on 30 May. After the week finishes, the first outcome report contains 100 jobs; a later correction adds 20 late-entered jobs, making the final outcome 120. Error against the first report is an overprediction of 10 jobs, while error against the final total is an underprediction of 10.
A retrospective model using corrected historical inputs produces 118, only two below the final total. That is not evidence that the live forecast was 118 or that those corrected inputs were available on 1 June. The analyst stores 110 as the actual historical forecast and labels 118 a separate retrospective reconstruction. The backtest also compares an eligible benchmark generated from the same 30 May cut-off. This preserves the distinction between improving a method with hindsight and measuring the decisions it could support at the time. Any historical period lacking a usable source snapshot is identified in the coverage summary, so the headline error does not imply a complete reconstruction.
TAKE THIS INTO YOUR NEXT CONVERSATION
A practical next-step checklist.
- Store event dates and availability dates separately.
- Freeze inputs, settings and forecast origin.
- Evaluate matching horizons against a stated outcome version.
- Compare with an equally constrained benchmark.
Common questions.
Can corrected outcomes be used to score forecasts?
Yes, if that outcome version matches the evaluation purpose and is clearly labelled. Corrected future inputs cannot be treated as historically available.
Does low historical error guarantee future accuracy?
No. Conditions, definitions and demand patterns can change, and model selection can overfit the evaluation window.
Further reading & source context.
These primary references provide background. They are not evidence for the fictional example above and do not endorse Accuralytix.
Found an error or a missing qualification? Send an editorial correction.
Check my data readiness.
Use a practical tool to turn this idea into a clearer starting point.
Check my data readiness