Forecasting foundation models still need local baselines

Forecasting foundation models still need local baselines. I will not promote a pretrained forecaster into production merely because it can produce a plausible zero-shot curve. A forecast earns operational trust by beating a simple model on the series, horizon, and loss that matter—and by staying calibrated after the data changes. The baseline is not ceremonial. It is the control that tells me whether added model breadth creates local value.

I separate three forecasting paradigms because they make different claims. A local model learns one series; a global model learns across a collection of related series; a zero-shot model applies patterns learned elsewhere without fitting to the target history. Chronos, TimesFM, Moirai, and TimeGPT make the third path practical, while also inviting global-style reasoning about shared structure. None of that removes the local question: what signal exists in this series that a modest model can already capture?

My first comparison is therefore ARIMA and ETS, not another large model. They encode useful, inspectable assumptions about autocorrelation, trend, and seasonality. If a foundation model cannot improve on them, its complexity is not buying forecast quality. The LTSF-Linear challenge made the broader lesson hard to ignore—simple linear models exposed weaknesses in transformer approaches to long-term series forecasting. Architecture prestige is not evidence against a relevant baseline.

The comparison must share a data contract. I apply the same normalization policy and cutoff discipline, then make lag, rolling, calendar, and Fourier features explicit where the model interface permits them. Otherwise I may be comparing preprocessing choices rather than forecasting ability. This is why I treat AI-ready data as modeled—the useful representation is an engineered decision, not a raw timestamp-value pair handed to every model indiscriminately.

I also refuse to collapse evaluation into one point-error number. MASE lets me read error relative to a simple scale-aware reference. CRPS and WQL test the quality of probabilistic forecasts rather than only their center. Interval coverage checks whether stated uncertainty contains observations at the expected rate. Together, these measures reveal a production distinction: a model can improve its central forecast while making its uncertainty less dependable.

I turn that distinction into an acceptance contract. The candidate must outperform ARIMA or ETS at the deployment horizon, across the series segments that drive decisions, while meeting a declared interval-coverage requirement. The exact gate belongs to the use case, not to the model family. That is evals as contract applied to forecasting—the launch rule is fixed before attractive examples tempt me to move it.

Deployment does not end the comparison. Concept drift can change level, seasonality, variance, or the relationship between recent history and the next outcome. I keep the local baseline running beside the foundation model and monitor their relative error and probabilistic behavior. A widening gap is often more actionable than either score alone because it shows whether the sophisticated model is losing specifically against current local structure. In that sense, drift is the production eval.

I make one precise concession: when a target series is short or has sparse history, a zero-shot foundation model may be the stronger starting point because a local ARIMA or ETS fit has little evidence to estimate from. Even there, I keep the baseline in the harness as observations accumulate. Conformal prediction can help recalibrate intervals against recent residuals, but it does not excuse drift monitoring or turn a weak point forecast into a strong one.

The production pattern is deliberately asymmetric. Foundation models compete for promotion; local baselines retain their place as controls. I version normalization and feature logic, evaluate point and distributional behavior, watch coverage, and rerun the contest as the series changes. That discipline keeps any foundation model from becoming an assumption embedded in infrastructure. The model may be broad, but the burden of proof remains local—and repeatable.