Multi-task learning with Multi-gate Mixture-of-experts

Multi-task learning is a machine learning method in which a model learns to solve multiple tasks simultaneously. The assumption is that by learning to complete multiple correlated tasks with the same model, that the performance of each task will be higher than if we trained individual models on each task. However, this assumption does not always hold true. Naïve multi-task learning approaches do not consider the relationships between tasks and trade-offs involved in learning to complete all of the tasks. Google's multi-gate mixture-of-experts model (MMoE) attempts to improve upon the baseline multi-task learning methods by explicitly learning relationships between tasks.