Existing research on continual learning of a sequence of tasks focused on dealing with catastrophic forgetting, where the tasks are assumed to be dissimilar and have little shared knowledge. Some work has also been done to transfer previously learned knowledge to the new task when the tasks are similar and have shared knowledge.

Weaknesses: * The dataset choice seems arbitrary. Since authors are defining a new setting, they should elaborate why specifically FEMNIST and FCelebA are used to create similar and dissimilar pairs. NeurIPS'19 also propose a similar masking based approach to learn non-overlapping paths for dissimilar tasks. These papers should be cited and disucssed (preferably compared against) in this manuscript. E.g., the prior works on task incremental learning have both sets of similar and dissimilar tasks.

Existing research on continual learning of a sequence of tasks focused on dealing with catastrophic forgetting, where the tasks are assumed to be dissimilar and have little shared knowledge. Some work has also been done to transfer previously learned knowledge to the new task when the tasks are similar and have shared knowledge. To the best of our knowledge, no technique has been proposed to learn a sequence of mixed similar and dissimilar tasks that can deal with forgetting and also transfer knowledge forward and backward. This paper proposes such a technique to learn both types of tasks in the same network. For dissimilar tasks, the algorithm focuses on dealing with forgetting, and for similar tasks, the algorithm focuses on selectively transferring the knowledge learned from some similar previous tasks to improve the new task learning.