Weaknesses: * The dataset choice seems arbitrary. Since authors are defining a new setting, they should elaborate why specifically FEMNIST and FCelebA are used to create similar and dissimilar pairs. NeurIPS'19 also propose a similar masking based approach to learn non-overlapping paths for dissimilar tasks. These papers should be cited and disucssed (preferably compared against) in this manuscript. E.g., the prior works on task incremental learning have both sets of similar and dissimilar tasks.

Humans can continuously learn new knowledge as their experience grows. In contrast, previous learning in deep neural networks can quickly fade out when they are trained on a new task. In this paper, we hypothesize this problem can be avoided by learning a set of generalized parameters, that are neither specific to old nor new tasks. In this pursuit, we introduce a novel meta-learning approach that seeks to maintain an equilibrium between all the encountered tasks. This is ensured by a new meta-update rule which avoids catastrophic forgetting. In comparison to previous meta-learning techniques, our approach is task-agnostic. When presented with a continuum of data, our model automatically identifies the task and quickly adapts to it with just a single update. We perform extensive experiments on five datasets in a class-incremental setting, leading to significant improvements over the state of the art methods (e.g., a 21.3% boost on CIFAR100 with 10 incremental tasks). Specifically, on large-scale datasets that generally prove difficult cases for incremental learning, our approach delivers absolute gains as high as 19.1% and 7.4% on ImageNet and MS-Celeb datasets, respectively.