AAAI Publications, Third AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Modeling Temporal Crowd Work Quality with Limited Supervision
Hyun Joon Jung, Matthew Lease

Last modified: 2015-09-23


While recent work has shown that a worker’s performance can be more accurately modeled by temporal correlation in task performance, a fundamental challenge remains in the need for expert gold labels to evaluate a worker’s performance. To solve this problem, we explore two methods of utilizing limited gold labels, initial training and periodic updating. Furthermore, we present a novel way of learning a prediction model in the absence of gold labels with uncertaintyaware learning and soft-label updating. Our experiment with a real crowdsourcing dataset demonstrates that periodic updating tends to show better performance than initial training when the number of gold labels are very limited (< 25).


crowdsourcing; human computation; prediction; uncertainty-aware learning; time-series modeling

Full Text: PDF