AAAI Publications, Second AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Viability of Crowd-Volunteered Open Research Reviews
Malay Bhattacharyya

Last modified: 2014-10-14


In this study, we examine the feasibility of reviewing research publications through crowdsourcing. We propose a post-publication review model where the research materials are posted online for the crowd volunteers to review. One such platform, named PuGLi, is prepared and the data collected from it is analyzed. The analysis raises some interesting questions related to the challenges of attracting crowd volunteers, determining truthfulness of opinions, and obtaining the final judgment from multiple opinions received from the crowd.


Crowdsourcing; Publication model; Computer supported cooperative work


1. Beaudouin-Lafon, M. 2012. Open access to scientific publications. Communications of the ACM 53(2):32–34.

2. Cross, A.; Bayyapunedi, M.; Ravindran, D.; Cutrell, E.; and Thies, W. 2014. VidWiki: Enabling the Crowd to Improve the Legibility of Online Educational Videos. In Proceedings of the CSCW, 1167–1175. ACM Press.

3. Kittur, A.; Nickerson, J. V.; Bernstein, M. S.; Gerber, E. M.; Shaw, A.; Zimmerman, J.; Lease, M.; and Horton, J. J. 2013.The Future of Crowd Work. In Proceedings of the CSCW, 1301–1318. ACM Press.

4. Terry, D. 2014. Publish Now, Judge Later. Communications of the ACM 57(1):44–46.

Full Text: PDF