A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers.

In this paper we consider an alternative strategy.

In many scenarios, this failure can be attributed to obscuring the crucial interplay between the training algorithm and the underlying data distribution.

Existing regularization methods merely minimize the norms of embeddings to regularize the model, leading to suboptimal performance.