AAAI Publications, Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence

Font Size: 
Beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks
Yu-An Sun, Shourya Roy, Greg Little

Last modified: 2011-08-24

Abstract


Quality assurance remains a key topic in human computation research field. Prior work indicates independent agreement is effective for low difficulty tasks, but has limitations. This paper addresses this problem by proposing a tournament selection based quality control process. The experimental results from this paper show that the human are better at identifying the correct answers than producing them themselves.

Full Text: PDF