Multi-horizon Probabilistic Forecasting with Conformal Prediction and NeuralProphet
In my previous articles “Benchmarking Neural Prophet. Part I — Neural Prophet vs Facebook Prophet” and “Benchmarking Neural Prophet. Part II — exploring electricity dataset”, we have looked at NeuralProphet and evaluated whether it delivers performance improvements vis-a-vis Facebook Prophet (TLDR; yes) and whether it generally delivers good point forecasting performance (TLDR; yes).
NeuralProphet is a much better model for point forecasting as it has addressed the principal issue of its predecessor Facebook Prophet — the inability of Facebook Prophet to model local dependencies using autoregressive (AR) terms. NeuralProphet effectively leverages the ability to model local patterns, which is essential for forecasting the near-term future.
The NeuralProphet core development team has recently released an exciting upgrade. NeuralProphet now includes state-of-the-art innovation in probabilistic forecasting, including probabilistic time-series forecasting using Conformal Prediction.
“Our [NeuralProphet core devs] goal is to accelerate the R&D of conformal prediction into the time-series space across any domain using NeuralProphet as the rapid experimentation and prototyping tool. Our current implementation of conformal prediction is only the beginning of our journey into conformal prediction. We invite our friends at the Conformal Prediction community to collaborate and co-develop with us to make the latest conformal prediction techniques more accessible to users in the time-series settings.”
If you are unfamiliar with Conformal Prediction, Awesome Conformal Prediction can help you get started in this very popular field. Conformal Prediction is the best uncertainty quantification framework for the XXIst century that allows model and data distribution free uncertainty estimation for any sample size and converts any statistical, machine or deep learning model into a probabilistic prediction model.
Whilst Conformal Prediction has been originally designed for IID data, the recent research and innovation in Conformal Prediction extended it into time series and forecasting. For the first time in the history of time series and forecasting, Conformal Prediction unlocked robust probabilistic forecasting that can produce Prediction Intervals with specified prediction coverage at any selected confidence level.
Before Conformal Prediction was extended to time series, most forecasting models could not produce correct Prediction Intervals. Most of the existing forecasting models either cannot produce Prediction Intervals or produce Prediction Intervals that are misspecified in terms of required coverage matching the specified confidence levels.
Using miscalibrated prediction intervals results in flawed decision making and negative business outcomes such as over/underestimating demand, under/over producing electricity by utilities — hence for any business it is essential to ensure that forecasting models produce correct prediction intervals.
Research has shown that many forecasting models (including statistical, machine and deep learning models of every kind) fail to estimate uncertainty properly.
In my previous article, “Probabilistic Forecasting with Conformal Prediction and NeuralProphet”, we looked at the recent release of NeuralProphet, which includes several options for probabilistic forecasting using Conformal Prediction, including implementing one of the most popular models Conformalized Quantile Regression (CQR). If you are unfamiliar with CQR, you can read about it in my Medium article “How to predict quantiles in a more intelligent way (or ‘Bye-bye quantile regression, hello Conformalized Quantile Regression’)”.
TLDR; CQR adjusts predictive intervals dynamically to account for local uncertainty in predictions. Unlike quantile regression that produces biased forecasts, CQR delivers well-calibrated prediction intervals by default regardless of the data distribution, the prediction model and the sample size.
In this article, we follow on from the previous article to look at NeuralProphet version 0.5.1, which has just been released. In this version, NeuralProphet received an exciting upgrade to its powerful probabilistic forecasting functionality extending the ability to produce multi-step probabilistic forecasts using Conformal Prediction.
We can now train NeuralProphet to produce multi-step probabilistic forecasts using Conformal Prediction, including one of the most popular models Conformalized Quantile Regression (CQR).
# specify quantiles
quantile_lo, quantile_hi = 0.05, 0.95
quantiles = [quantile_lo, quantile_hi]
n_lags = 7
n_forecasts = 7
m_multistep_quantile_regression = NeuralProphet(
n_forecasts=n_forecasts,
growth='off',
yearly_seasonality=False,
weekly_seasonality=False,
daily_seasonality=False,
n_lags=n_lags,
learning_rate=0.01,
quantiles = quantiles
)
%%time
random_seed = 0
# Concatenate train_df and val_df as full training set
set_random_seed(random_seed)
metrics_quantile = m_multistep_quantile_regression.fit(pd.concat([train_df,
val_df]), freq="D")Forecasting on the test set, we now obtain a probabilistic forecast for seven days ahead using classical quantile regression.
Are we done then yet? Not quite — remember, classical quantile regression does not produce calibrated Prediction Intervals. Just because we specified 5% and 95% quantiles as a wish list does not mean that 90% of test points will be within this Prediction Interval.
This is where the magic of Conformal Prediction comes in — we can correct Prediction Intervals produced by any model, including the quantile regression we trained so far with NeuralProphet.
We now complete the last step — “conformalize” our probabilistic forecasting model.
All we need to do is to call .conformal_predict() method on our quantile regression model; we use a calibration set that was set aside. This step aims to correct Prediction Intervals created by quantile regression so that they can deliver declared coverage in line with the selected 90% confidence level.
method = "naive"
alpha = 0.1
plotting_backend = "plotly" # "plotly", None
naive_multistep_forecast_conformal_test_df = m_multistep_quantile_regression.conformal_predict(
test_df,
calibration_df=cal_df,
alpha=alpha,
method=method,
plotting_backend=plotting_backend,
)
As part of the results, NeuralProphet helpfully produces a plot showing how the width of the selected interval (y_hat-q1, y_hat+q1) varies according to how many steps ahead we forecast. The width of the prediction interval reflects many factors, including electricity consumption patterns during the week.
In the plot above, we observe how Conformal Prediction corrected predictions from NeuralProphet.
We now use more powerful method CQR (Conformalized Quantile Regression).
method = "cqr"
alpha = 0.1
plotting_backend = "plotly" # "plotly", None
cqr_multistep_forecast_conformal_test_df = m_multistep_quantile_regression.conformal_predict(
test_df,
calibration_df=cal_df,
alpha=alpha,
method=method,
plotting_backend=plotting_backend,
)
In the plot above, we have explored how forecasting one step ahead works; lets now look at seven days ahead forecasts.
As we can see, CQR has successfully provided prediction intervals for 7 days ahead forecasts.
Finally we can evaluate and compare performance of the naive method correction with that of CQR using validity (coverage) and efficiency (interval width) as the criteria. You can read more about evaluation of probabilistic forecasts in my Medium article “How to evaluate Probabilistic Forecasts”
We can see that CQR has basically halved miscoverage rate compared to the naive method resulting in better forecasts.
References:
- “Probabilistic Forecasting with Conformal Prediction and NeuralProphet”
- “Benchmarking Neural Prophet. Part I — Neural Prophet vs Facebook Prophet.”
- “Benchmarking Neural Prophet. Part II — exploring electricity dataset”
- “The M4 Forecasting Competition Prediction Intervals”
- How to predict quantiles in a more intelligent way (or ‘Bye-bye quantile regression, hello Conformal Quantile Regression).
- https://pypi.org/project/neuralprophet/
- Conformal Prediction using Energy Hospital Load tutorial
