🚨 Forecast Evaluation, Cross-Validation, and the Hidden Leakage Problem

When evaluating forecasting models, most practitioners use metrics like RMSE or MAE. But competitions like M4 and M5 popularized scaled error metrics such as MASE (Mean Absolute Scaled Error) and RMSSE (Root Mean Squared Scaled Error). These have a big advantage: they normalize errors by the typical variation in each series, making scores comparable across series with different scales.

So far so good. But what happens when you want to use cross-validation (CV) with these metrics?

This is where things get subtle — and where you might accidentally introduce data leakage into your evaluation.

When evaluating forecasting models, most practitioners use metrics like RMSE or MAE. But competitions like M4 and M5 popularized scaled error metrics such as MASE (Mean Absolute Scaled Error) and RMSSE (Root Mean Squared Scaled Error). These have a big advantage: they normalize errors by the typical variation in each series, making scores comparable across series with different scales.

So far so good. But what happens when you want to use cross-validation (CV) with these metrics?

This is where things get subtle — and where you might accidentally introduce data leakage into your evaluation.

MASE is the MAE divided by the mean absolute lag-mm difference in the training data:

RMSSE is the same idea, but with squared errors: