Jackknife+ — a Swiss knife of Conformal Prediction for regression

A while back in my article ‘How to predict quantiles in a more intelligent way (or ‘Bye-bye quantile regression, hello Conformalized Quantile Regression’) I have covered Conformalized Quantile Regression — one of the most popular Conformal Prediction models. Today, we will look at another powerful Conformal Prediction model for quantifying uncertainty in regression — the Jackknife+.

Jackknife+ is a powerful Conformal Prediction method that was developed by the leading machine learning researchers from University of Chicago, Stanford, Carnegie Mellon and Berkeley and published in the paper ‘Predictive inference with the jackknife+’.

The jackknife technique was originally conceived by Maurice Quenouille between 1949 and 1956. Later in 1958, John Tukey, a renowned statistician, further refined this technique. He proposed the term ‘jackknife’ as a metaphor for this method due to its wide applicability and versatility. Much like a physical jackknife — a compact, foldable knife — this statistical method offers a flexible and adaptable solution to a broad spectrum of problems, despite the existence of other tools that might solve specific problems more efficiently.

Our objective is to establish a regression function using the training data, which consists of feature pairs (Xi, Yi). We seek to predict the output Yn+1 for a new feature vector Xn+1=x and generate a respective uncertainty interval for this prediction. The intent is for this interval to include the true Yn+1 with a predefined coverage probability.

A straightforward approach might be to fit the underlying regression model to the training data, compute the residuals, and then use these residuals to estimate the quantile. This quantile could then be used to determine the width of the prediction interval for the new test point. However, this method tends to underestimate the actual uncertainty because of overfitting — the residuals derived from the training set are usually smaller than those you would get from unseen test data.

To mitigate the issue of overfitting, a robust statistical technique named ‘jackknife’ was devised. The original purpose of this technique was to reduce bias and provide an estimate for variance. It operates on the principle of sequentially excluding one observation from the dataset and re-estimating the model. This methodology provides an empirical means to assess the stability of the model and its…