Reviews: Multi-Task Learning as Multi-Objective Optimization

Overall summary of the paper: This paper proposed a multi-task learning algorithm from multi-objective optimization perspective and the authors provided an approximation algorithm, which could accelerate the training process. The authors claim that existing MTL algorithms used linear combinations (uniform weight) of the loss from different tasks, which is hard to achieve the Pareto optimality. Unlike the uniform weight strategy, the authors use the MGDA algorithm to solve the optimal weight, which would increase the performance for all the tasks to achieve the Pareto optimality. Moreover, when solving the sub-problem for shared parameters, the author gave an upper bound of the loss function, and this upper bound optimization only requires one-time back propagation regardless of the number of tasks. The results show that the approximation strategy not only can accelerate the training process but also improve the performance.