Mixture-of-Experts (MoE) models have become a key approach for scaling large language models efficiently by activating only a subset of experts during training and inference. Typically, the number of activated experts presents a trade-off: fewer experts reduce computational costs, while more experts improve performance. Recent studies reveal that not all activated experts contribute equally to model performance, with some providing minimal utility, particularly when finetuning pretrained MoE models for specialized downstream tasks. The co-existence of significant and redundant parameters in experts provides us an opportunity to reduce the number of activated experts while maintaining model performance. In this work, we propose the concept of compressed experts, lightweight modules that serve as compact representations of full experts. Our approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts. The reduction of active parameters significantly lowers inference costs while achieving comparable performance. Extensive experiments on models including Phi-MoE and OLMoE demonstrate that compressed experts recover over 90% of full expert performance across various tasks while reducing more than 30% active parameters and saving 20% in inference costs. This approach enables efficient deployment of MoE models in resource-constrained settings and facilitates scaling to larger models with manageable overhead. Our code is available at https://github.com/yifei-he/Compressed-Experts.

For the prediction with experts' advice setting, we consider some methods to construct forecasting algorithms that suffer loss not much more than any expert in the pool. In contrast to the standard approach, we investigate the case of long-term forecasting of time series. This approach implies that each expert issues a forecast for a time point ahead (or a time interval), and then the master algorithm combines these forecasts into one aggregated forecast (sequence of forecasts). We introduce two new approaches to aggregating experts' long-term interval predictions. Both are based on Vovk's aggregating algorithm. The first approach applies the method of Mixing Past Posteriors method to the long-term prediction. The second approach is used for the interval forecasting and considers overlapping experts. The upper bounds for regret of these algorithms for adversarial case are obtained. We also present the results of numerical experiments on time series long-term prediction.