My master's thesis focused on air cargo demand forecasting. The lasting lesson is not that one model family always wins. It is that forecasting becomes useful only when the target, the evaluation, and the delivery process match the decision somebody needs to make.
The examples below illustrate that problem rather than reproduce a confidential dataset or claim new thesis results. Cargo demand is a helpful setting because the distance between a good statistical score and a useful planning input is easy to see.
Define the forecast around a decision
“Predict demand” leaves several important choices unresolved. At what level: total network, route, or customer? At what horizon? When must the prediction be available? What can a planner still change after receiving it?
A four-week capacity decision and a next-day staffing decision need different targets and validation. Write those choices down before comparing architectures. Otherwise a model can win an experiment that never represented the operational task.
Make the baseline difficult to dismiss
Compare with simple alternatives that a planning team could actually operate. A seasonal-naive forecast repeats the corresponding observation from a previous season. Adding a trend adjustment creates a different baseline and should be named and tested separately.
Baselines reveal how much improvement comes from modeling rather than from the predictability already present in the data. Compare their maintenance needs, failure behavior, and explanation cost as well as their errors. A small average gain does not automatically justify a more fragile deployment.
Use only information available at prediction time
Calendar information can be known in advance; revised economic data may not have been available when the forecast was issued. External regressors need an availability timestamp and a plan for obtaining their future values. Feeding actual future observations into a historical test makes the task easier than production.
Fit preprocessing within each training window. Record the data vintage and forecasting cutoff. These details are less glamorous than the model, but they determine whether a backtest is answering an honest question.
Evaluate across multiple forecast origins
Rolling-origin evaluation moves the forecast cutoff through time and scores the horizons that matter. It can show whether improvements persist across seasons and changing conditions. It does not guarantee robustness to a shock that the observed history never contained.
Inspect errors by horizon and route rather than only at the network level. MAPE needs particular care when actual volumes are zero or small. Absolute errors, scaled measures, and decision-specific costs each reveal different aspects of performance; no single metric removes the need to examine cases.
Connect uncertainty to the handoff
A planning interface should identify the forecast issue date, horizon, data freshness, and limitations. If intervals are supplied, evaluate their coverage and width on held-out periods. A very wide interval can achieve reassuring coverage without being useful for a capacity decision.
Define what happens when a feed is late or the forecast job fails. A clearly labeled fallback may be more useful than silently displaying yesterday's output as today's. Record manual overrides with reasons so they can be reviewed without treating every override as ground truth.
The lesson beyond forecasting
An ML project is not complete when it writes a prediction file. It is complete enough to operate when somebody owns the inputs, understands the errors, receives the result at the right time, and knows how to recover when it fails. That same pattern appears in data contracts and product delivery.
Match the evaluation unit to the planning unit
Keep route-level and aggregate errors separate. A low network error can arise when over- and under-predictions cancel across routes that still need individual capacity decisions. Compare horizon-specific losses and any business cost function at the level where the planner acts. For dependent forecast errors across successive origins, a conventional confidence interval assuming independent errors can be misleading; choose a time-aware uncertainty comparison and describe its assumptions.