Progression in a Language Annotation Game with a Purpose

  • Chris Madge Queen Mary University of London
  • Juntao Yu Queen Mary University of London
  • Jon Chamberlain University of Essex
  • Udo Kruschwitz University of Regensburg
  • Silviu Paun Queen Mary University of London
  • Massimo Poesio Queen Mary University of London

Abstract

Within traditional games design, incorporating progressive difficulty is considered of fundamental importance. But despite the widespread intuition that progression could have clear benefits in Games-With-A-Purpose (GWAPs)–e.g., for training non-expert annotators to produce more complex judgements– progression is not in fact a prominent feature of GWAPs; and there is even less evidence on its effects. In this work we present an approach to progression in GWAPs that generalizes to different annotation tasks with minimal, if any, dependency on gold annotated data. Using this method we observe a statistically significant increase in accuracy over randomly showing items to annotators.

Published
2019-10-28