Adaptive Attention Span in Transformers

Part of its success is due to its ability to model called Sequential Transformer capture long term dependencies. This is achieved (Vaswani et al., 2017). A Transformer is by taking long sequences as inputs and explicitly made of a sequence of layers that are composed of compute the relations between every token via a a block of parallel self-attention layers followed mechanism called the "self-attention" layer (Al-by a feedforward network. We refer to Vaswani Rfou et al., 2019).