Representational Strengths and Limitations of Transformers

In recent years, transformer networks [V aswani et al., 2017] have been established as a fundamental neural architecture powering state-of-the-art results in many applications, including language