A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention

Such investigations have demonstrated that attention layers are able to implement a wide range of different algorithms, even for the same task, using both positional and semantic attributes of the inputs.