Algorithmic Capabilities of Random Transformers

Why is this the case? One possibility is that some aspect of the transformer architecture makes these behaviors easy to learn. Under this hypothesis, transformer models do not implement any useful functionality when initialized; however, their loss landscape is structured such that they can be (computation-and sample-) efficiently optimized for behaviors of interest.