AAAI Publications, Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence

Font Size: 
MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture (Extended Abstract)
Anand Kulkarni, David Rolnitzky, Philipp Gutheim, Prayag Narula, Tapan Parikh, Bjoern Hartmnn

Last modified: 2012-07-15


Online labor marketplaces offer the potential to automate a variety of tasks too difficult for computers, but present requesters with significant difficulties in obtaining accurate results. We share experiences from building MobileWorks, a crowd platform that departs from the marketplace model to provide robust, high-quality results. Three architectural contributions yield measurably improved accuracy on input tasks.  A dynamic work routing system identifies expertise in the crowd and ensures that all work posted into the system is completed with bounded completion times and at fair worker prices. A peer management system ensures that incorrect answers are prevented by experienced members of the crowd. Last, social interaction techniques give the best workers the ability and incentives to manage, teach & supervise other members of the crowd, as well as to clarify tasks. This process filters worker error and allows the crowd to collectively learn how to solve unfamiliar tasks. (extended abstract)

Full Text: PDF