Transformers on Markov data: Constant depth suffices

Attention-based transformers have been remarkably successful at modeling generative processes across various domains and modalities.