AAAI Publications, Second AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Tuning the Diversity of Open-Ended Responses From the Crowd
Walter S. Lasecki, Christopher M. Homan, Jeffrey L. Bigham

Last modified: 2014-09-05

Abstract


Crowdsourcing can solve problems beyond the reach of state-of-the-art fully automated systems. A common pattern found in many such systems is for the workers to discover, in parallel, a number of candidate solutions and then vote on the best one to pass forward, often within a fixed amount of time. We present the propose-vote-abstain mechanism for eliciting from crowd workers the proper balance between solution discovery and selection. Each crowd worker is given a choice among proposing an answer, voting among the answers proposed so far, or abstaining, i.e., doing nothing. When a stopping condition is reached, the mechanism returns the answer with the most votes. Workers are paid a base amount, with bonuses if they propose or vote for the winning answer.

Keywords


Microtasks; Game Theory

References


Aral, S.; Ipeirotis, P.; and Taylor, S. 2011. Content and context: Identifying the impact of qualitative information on consumer choice. In Galletta, D. F., and Liang, T.-P., eds., ICIS. Association for Information Systems.

Attenberg, J.; Ipeirotis, P. G.; and Provost, F. J. 2011. Beat the machine: Challenging workers to find the unknown unknowns. In Human Computation, volume WS-11-11 of AAAI Workshops. AAAI.

Bernstein, M. S.; Brandt, J.; Miller, R. C.; and Karger, D. R. 2011. Crowds in Two Seconds: Enabling Realtime Crowd- powered Interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2011), 33–42.

Bigham, J. P.; Jayant, C.; Ji, H.; Little, G.; Miller, A.; Miller, R. C.; Miller, R.; Tatarowicz, A.; White, B., W. S.; and Yeh, T. 2010. VizWiz: nearly real-time answers to visual ques- tions. In Proceedings of the ACM Symposium on User Inter- face Software and Technology (UIST 2010), 333–342.

Lasecki, W. S.; Murray, K. I.; White, S.; Miller, R. C.; and Bigham, J. P. 2011. Real-time Crowd Control of Existing Interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2011), 23–32.

Lasecki, W. S.; Miller, C. D.; Sadilek, A.; Abu- Moussa, A.; and Bigham, J. 2012. Real-time cap- tioning by groups of non-experts. In In Submission. http://hci.cs.rochester.edu/pubs/pdfs/legion-scribe.pdf.

von Ahn, L., and Dabbish, L. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, CHI ’04, 319–326. New York, NY, USA: ACM.


Full Text: PDF