AAAI Publications, Third AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Reliable Aggregation of Boolean Crowdsourced Tasks
Luca de Alfaro, Vassilis Polychronopoulos, Michael Shavlovsky

Last modified: 2015-09-23


We propose novel algorithms for the problem of crowdsourcing binary labels. Such binary labeling tasks are very common in crowdsourcing platforms, for instance, to judge the appropriateness of web content or to flag vandalism. We propose two unsupervised algorithms: one simple to implement albeit derived heuristically, and one based on iterated bayesian parameter estimation of user reputation models. We provide mathematical insight into the benefits of the proposed algorithms over existing approaches, and we confirm these insights by showing that both algorithms offer improved performance on many occasions across both synthetic and real-world datasets obtained via Amazon Mechanical Turk.


iterative methods; spam; reputation systems; aggregation

Full Text: PDF