Review for NeurIPS paper: AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

Weaknesses: Despite its effectiveness, there are certain aspects that I am concerned about, as following: (1) The idea of layer dropping is not new. It has been explored for regularization [1] as well as structured pruning [2, 3]. In addition, methods of routing subnetwork and learning task-specific params for each task [4] have also been studied before. It would be more comprehensive to additionally test on a dataset with larger numbers of tasks. First, I think that the method's main advantage is not memory efficiency since it is at least as large as a standard MTL network (denoted as Multi-Task in the paper).