AAAI Publications, Third AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Reliable Aggregation of Boolean Crowdsourced Tasks
Luca de Alfaro, Vassilis Polychronopoulos, Michael Shavlovsky

Last modified: 2015-09-23

Abstract


We propose novel algorithms for the problem of crowdsourcing binary labels. Such binary labeling tasks are very common in crowdsourcing platforms, for instance, to judge the appropriateness of web content or to flag vandalism. We propose two unsupervised algorithms: one simple to implement albeit derived heuristically, and one based on iterated bayesian parameter estimation of user reputation models. We provide mathematical insight into the benefits of the proposed algorithms over existing approaches, and we confirm these insights by showing that both algorithms offer improved performance on many occasions across both synthetic and real-world datasets obtained via Amazon Mechanical Turk.

Keywords


iterative methods; spam; reputation systems; aggregation

Full Text: PDF