AAAI Publications, Third AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
The ActiveCrowdToolkit: An Open-Source Tool for Benchmarking Active Learning Algorithms for Crowdsourcing Research
Matteo Venanzi, Oliver Parson, Alex Rogers, Nick Jennings

Last modified: 2015-09-23


We present an open-source toolkit that allows the easy comparison of the performance of active learning methods over a series of datasets. The toolkit allows such strategies to be constructed by combining a judgement aggregation model, task selection method and worker selection method.The toolkit also provides a user interface which allows researchers to gain insight into worker performance and task classification at runtime.


Crowdsourcing, Active learning, Benchmarking

Full Text: PDF