Curbing Task Interference using Representation Similarity-Guided Multi-Task Feature Sharing

Multi-task learning of dense prediction tasks, by sharing both the encoder and decoder, as opposed to sharing only the encoder, provides an attractive front to increase both accuracy and computational efficiency. When the tasks are similar, sharing the decoder serves as an additional inductive bias providing more room for tasks to share complementary information among themselves. However, increased sharing exposes more parameters to task interference which likely hinders both generalization and robustness. Effective ways to curb this interference while exploiting the inductive bias of sharing the decoder remains an open challenge. To address this challenge, we propose Progressive Decoder Fusion (PDF) to progressively combine task decoders based on inter-task representation similarity. We show that this procedure leads to a multi-task network with better generalization to in-distribution and out-of-distribution data and improved robustness to adversarial attacks. Additionally, we observe that the predictions of different tasks of this multi-task network are more consistent with each other. Code is made available at github.com/NeurAI-Lab/ Obtaining real-time predictions from neural networks is imperative for time-critical applications such as autonomous driving. These applications also require predictions from multiple tasks to shed light on varied aspects of the input scene. Multi-task networks (MTNs) can elegantly combine these two requirements by jointly predicting multiple tasks while sharing a considerable number of parameters among tasks. On the other hand, training separate single task networks could lead to different task predictions contradicting each other. Moreover, each of these networks has to be individually robust to various forms of adverse inputs such as image corruptions and adversarial attacks. MTNs include an inductive bias in the shared parameter space which encourages tasks to share complementary information to improve predictions.