AAAI Publications, First AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
EM-Based Inference of True Labels Using Confidence Judgments
Satoshi Oyama, Yukino Baba, Yuko Sakurai, Hisashi Kashima

Last modified: 2013-11-03


We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.


crowdsourcing; human computation; quality control; confidence judgment; EM algorithm

Full Text: PDF