AAAI Publications, Twenty-Ninth AAAI Conference on Artificial Intelligence

Font Size: 
Obtaining Well Calibrated Probabilities Using Bayesian Binning
Mahdi Pakdaman Naeini, Gregory Cooper, Milos Hauskrecht

Last modified: 2015-02-21

Abstract


Learning probabilistic predictive models that are well calibrated is critical for many prediction and decision-making tasks in artificial intelligence. In this paper we present a new non-parametric calibration method called Bayesian Binning into Quantiles (BBQ) which addresses key limitations of existing calibration methods. The method post processes the output of a binary classification algorithm; thus, it can be readily combined with many existing classification algorithms. The method is computationally tractable, and empirically accurate, as evidenced by the set of experiments reported here on both real and simulated datasets.

Keywords


Bayesian binning; classifier calibration; accurate probability; calibrated probability; Bayesian Scoring

Full Text: PDF