Mixture of Tokens: Continuous MoE through Cross-Example Aggregation

Most widely adopted MoE models are discontinuous with respect to their parameters - often referred to as sparse .