BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance compared to dense models. However, training MoEs from scratch in a large-scale regime is prohibitively expensive.