While some workshowsthatmulti-task networkstrained viaunitary scalarization exhibit superior performance to independent per-task models [29, 35], others suggest the opposite [30, 54, 58]. However, SMTOs usually require access to per-task gradients either with respect to the shared parameters, or to the shared representation.

The optimization landscape of each task consists of a deep valley, a property that has been observed in neural network optimization landscapes [22], and the bottom ofeachvalleyischaracterized by high positive curvature and large differences in the task gradient magnitudes.

To achieve this, we propose leveraging existing MTL methods, but the presence of a huge number of denoising tasks makes this computationally expensive to calculate the necessary per-task loss or gradient.