Context-Aware Self-Attention Networks

  • Baosong Yang University of Macau
  • Jian Li The Chinese University of Hong Kong
  • Derek F. Wong University of Macau
  • Lidia S. Chao University of Macau
  • Xing Wang Tencent AI Lab
  • Zhaopeng Tu Tencent AI Lab

Abstract

Self-attention model has shown its flexibility in parallel computation and the effectiveness on modeling both long- and short-term dependencies. However, it calculates the dependencies between representations without considering the contextual information, which has proven useful for modeling dependencies among neural representations in various natural language tasks. In this work, we focus on improving self-attention networks through capturing the richness of context. To maintain the simplicity and flexibility of the self-attention networks, we propose to contextualize the transformations of the query and key layers, which are used to calculate the relevance between elements. Specifically, we leverage the internal representations that embed both global and deep contexts, thus avoid relying on external resources. Experimental results on WMT14 English⇒German and WMT17 Chinese⇒English translation tasks demonstrate the effectiveness and universality of the proposed methods. Furthermore, we conducted extensive analyses to quantify how the context vectors participate in the self-attention model.

Published
2019-07-17
Section
AAAI Technical Track: AI and the Web