Microsoft's Tutel optimizes mixture of experts model training

Let the OSS Enterprise newsletter guide your open source journey! Microsoft this week announced Tutel, a library to support the development of mixture of experts (MoE) models -- a particular type of large-scale AI model. Tutel, which is open source and has been integrated into fairseq, one of Facebook's toolkits in PyTorch, is designed to enable developers across AI disciplines to "execute MoE more easily and efficiently," Microsoft says. MoE are made up of small clusters of "neurons" that are only active under special, specific circumstances. Lower "layers" of the MoE model extract features and experts are called upon to evaluate those features.