AAAI Publications, Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence

Font Size: 
Dynamically Switching between Synergistic Workflows for Crowdsourcing
Christopher H Lin, . Mausam, Daniel S Weld

Last modified: 2012-07-15


To ensure quality results from unreliable crowdsourced workers, task designers often construct complex workflows and aggregate worker responses from redundant runs. Frequently, they create several alternative workflows to accomplish the task, and choose a single workflow to deploy (perhaps the one that achieves the best performance during early experiments). However, this seemingly natural design paradigm does not achieve the full potential of crowdsourcing. In particular, using a single workflow (even the best) to accomplish a task is suboptimal. We show that alternative workflows can compose synergistically to yield a much higher quality output. We formalize the insight with a novel probabilistic graphical model, design and implement AgentHunt, a POMDP-based controller that dynamically switches between these workflows to achieve higher returns on investment, and design offline and online methods for learning model parameters. Live experiments on Amazon Mechanical Turk demonstrate the superiority of AgentHunt for the practical task of generating NLP training data, yielding up to 50% error reduction and greater net utility compared to previous methods.


crowdsourcing; artificial intelligence; decision-theory

Full Text: PDF