AAAI Publications, Second AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Behavior-Based Quality Assurance in Crowdsourcing Markets
Michael Feldman, Abraham Bernstein

Last modified: 2014-09-05


Quality assurance in crowdsourcing markets has appeared to be an acute problem over the last years. We propose a quality control method inspired by Statistical Process Control (SPC), commonly used to control output quality in production processes and characterized by relying on time-series data. Behavioral traces of users may play a key role in evaluating the performance of work done on crowdsourcing platforms. Therefore, in our experiment we explore fifteen behavioral traces for their ability to recognize the drop in work quality. Preliminary results indicate that our method has a high potential for real-time detection and signaling a drop in work quality.


crowdsourcing;quality assurance


Oleson, D., Sorokin, A., Laughlin, G. P., Hester, V., Le, J., and  Biewald, L. 2011. Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation, 11, 11.

Montgomery, D. C. 2007.  Introduction to Statistical Quality Control. John Wiley & Sons.

Rzeszotarski, J. M., and Kittur, A. 2011. Instrumenting the Crowd: Using implicit Behavioral Measures to Predict Task Performance. In Proceedings of the 24th annual ACM symposium on User interface software and technology, 13-22. ACM

Full Text: PDF