AAAI Publications, Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence

Font Size: 
Crowdsourcing Control: Moving Beyond Multiple Choice
Christopher H. Lin, Mausam Mausam, Daniel S Weld

Last modified: 2012-07-15


To ensure quality results from crowdsourced tasks requesters often aggregate worker responses and use one of a plethora of strategies for the process of inferring the correct answer from the set of noisy responses. However, all current models assume prior knowledge of all possible outcomes of the task. While not an unreasonable assumption for tasks that can be posited as multiple-choice questions (e.g. n-ary classification), we observe that many tasks do not naturally fit this paradigm, but instead demand a free-response, generalized, formulation where the outcome space is of infinite size (e.g. audio transcription). We call these tasks open questions. We model open questions with a novel probabilistic graphical model, and design and implement LazySusan, a decision-theoretic controller that dynamically requests responses as necessary in order to infer answers to these tasks. Live experiments on Amazon Mechanical Turk demonstrate the superiority of LazySusan at solving SAT Math questions, eliminating 83.2% of the error and achieving greater net utility compared to the state-of-the-art strategy, majority voting.


crowdsourcing; artificial intelligence; decision-theory

Full Text: PDF