AAAI Publications, Second AAAI Conference on Human Computation and Crowdsourcing

Font Size: 
Context Trees: Crowdsourcing Global Understanding from Local Views
Vasilis Verroios, Michael S. Bernstein

Last modified: 2014-09-05


Crowdsourcing struggles when workers must see all of the pieces of input to make an accurate judgment. For example, to find the most important scenes in a novel or movie, each worker must spend hours consuming the entire plot to acquire a global understanding and then apply that understanding to each local scene. To enable the crowdsourcing of large-scale goals with only local views, we introduce context trees, a crowdsourcing workflow for creating global summaries of a large input. Context trees recursively combine elements through written summaries to form a tree. Workers can then ground their local decisions by applying those summaries back down to the leaf nodes. In the case of scale ratings such as scene importance, we introduce a weighting process that percolates ratings downwards through the tree so that important nodes in unimportant branches are not overweighted. When using context trees to rate the importance of scenes in a 4000-word story and a 100-minute movie, workers’ ratings are nearly as accurate as those who saw the entire input, and much improved over the traditional approach of splitting the input into independent segments. To explore whether context trees enable crowdsourcing to undertake new classes of goals, we also crowdsource the solution to a large hierarchical puzzle of 462,000 interlocking pieces.


Extractive Summarization; Abstractive Summarization; Crowdsourcing Global Understanding; Crowd Local Views; Context trees


Andre ́, P.; Kittur, A.; and Dow, S. P. 2014. Crowd synthesis: extracting categories and clusters from complex data. In Proc. CSCW.

Bernstein, M. S.; Little, G.; Miller, R. C.; Hartmann, B.; Ackerman, M. S.; Crowell, D.; and Panovich, K. 2010. Soylent: a word processor with a crowd inside. In Proc. UIST.

Biswas, A., and Jacobs, D. W. 2012. Active image clustering: Seeking constraints from humans to complement algorithms. In Proc. CVPR.

Bragg, J.; Mausam; and Weld, D. S. 2013. Crowdsourcing multi-label classification for taxonomy creation. In Proc. HCOMP.

Chilton, L. B.; Little, G.; Edge, D.; Weld, D. S.; and Landay, J. A. 2013. Cascade: Crowdsourcing taxonomy creation. In Proc. CHI.

Gomes, R.; Welinder, P.; Krause, A.; and Perona, P. 2011. Crowdclustering. In Proc. NIPS.

Gupta, V., and Lehal, G. 2010. A survey of text summarization extractive techniques. Journal of Emerging Technologies in Web Intelligence.

Herlocker, J.; Konstan, J. A.; and Riedl, J. 2002. An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Journal of Information Retrieval.

Kittur, A.; Smus, B.; Khamkar, S.; and Kraut, R. E. 2011. Crowd-forge: Crowdsourcing complex work. In Proc. UIST.

Kittur, A.; Chi, E. H.; and Suh, B. 2008. Crowdsourcing user studies with mechanical turk. In Proc. CHI.

Kulkarni, A.; Can, M.; and Hartmann, B. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proc. CSCW.

Lasecki, W.; Miller, C.; Sadilek, A.; Abumoussa, A.; Borrello, D.; Kushalnagar, R.; and Bigham, J. 2012a. Real-time captioning by groups of non-experts. In Proc. UIST.

Lasecki, W. S.; White, S.; Murray, K. I.; and Bigham, J. P. 2012b. Crowd memory: Learning in the collective. In Proc. Collective Intelligence.

Lasecki, W. S.; Wesley, R.; Nichols, J.; Kulkarni, A.; Allen, J. F.; and Bigham, J. P. 2013. Chorus: a crowd-powered conversational assistant. In Proc. UIST.

Norton, M. I.; Mochon, D.; and Ariely, D. 2012. The {IKEA} effect: When labor leads to love. Journal of Consumer Psychology.

Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proc. KDD.

Tamuz, O.; Liu, C.; Belongie, S.; Shamir, O.; and Kalai, A. 2011. Adaptively learning the crowd kernel. In Proc. ICML.

Wang, J.; Li, G.; Kraska, T.; Franklin, M. J.; and Feng, J. 2013. Leveraging transitive relations for crowdsourced joins. In Proc. SIGMOD.

Whang, S. E.; Lofgren, P.; and Garcia-Molina, H. 2013. Question selection for crowd entity resolution. In Proc. VLDB.

Zhang, H.; Law, E.; Miller, R.; Gajos, K.; Parkes, D.; and Horvitz, E. 2012. Human computation tasks with global constraints. In Proc. CHI.

Full Text: PDF