How to calibrate your classifier in an intelligent way using Venn-Abers Conformal Prediction

Venn-Abers predictors are the new kid on the block — and anyone can run them with just a few lines of Python code

Machine learning classification tasks are pervasive, ranging from differentiating between cats and dogs to diagnosing severe diseases and identifying pedestrians for self-driving cars.

However, these problem definitions often overshadow the true goal of classification: facilitating informed decision-making. Merely having class labels doesn’t suffice; what’s crucial are well-calibrated class probabilities.

While many data scientists gauge a model’s efficacy using metrics like accuracy, precision, and recall, these can be misleading outside of basic scenarios like the cats-vs-dogs example. Regrettably, essential topics like classifier calibration often go unaddressed in foundational machine learning courses, such as Andrew Ng’s renowned machine learning course.

For researchers, data scientists in the corporate realm, or machine learning engineers developing crucial applications, classifier calibration should be a top concern. One might wonder, ‘Why fuss over classifier calibration? Isn’t it simpler to just classify and move on?’

The answer is more nuanced. At the heart of classification lies the goal of fostering informed decision-making. These decisions inherently involve weighing the probabilities of various choices, along with the benefits and costs tied to each option.

“The primary goal of addressing a classification problem is to facilitate informed decision-making. Such decisions encompass the likelihoods of various available choices, and the corresponding advantages and drawbacks of each. Consider a scenario where a bank deliberates on granting a business loan. If you develop a classifier that merely states a prospective client won’t default, is that genuinely helpful? Not in the slightest.

For making a sound decision, especially with potentially millions on the line, businesses require models that provide accurate probabilities of a client defaulting or repaying the loan. These probability estimates can be integrated with the financial gains or losses from different outcomes to compute the net present value (NPV) of the loan decision.

However, the challenge is that many machine learning models don’t genuinely yield class probabilities. At…