An Application of Transfer to American Football: From Observation of Raw Video to Control in a Simulated Environment

  • David J. Stracuzzi Sandia National Laboratories
  • Alan Fern Oregon State University
  • Kamal Ali Stanford University
  • Robin Hess Oregon State University
  • Jervis Pinto Oregon State University
  • Nan Li Carnegie Mellon University
  • Tolga Konik Stanford University
  • Daniel G. Shapiro Institute for the Study of Learning and Expertise


Automatic transfer of learned knowledge from one task or domain to another offers great potential to simplify and expedite the construction and deployment of intelligent systems. In practice however, there are many barriers to achieving this goal. In this article, we present a prototype system for the real-world context of transferring knowledge of American football from video observation to control in a game simulator. We trace an example play from the raw video through execution and adaptation in the simulator, highlighting the system's component algorithms along with issues of complexity, generality, and scale. We then conclude with a discussion of the implications of this work for other applications, along with several possible improvements.