BeyondNot-Forgetting: ContinualLearningwith BackwardKnowledgeTransfer

Forexample, regularization-based methods (e.g., [12,1,18]) penalize the modification of important weights of oldtasks; parameter-isolation based methods (e.g., [7,26,31,9])fixthemodel learnt foroldtasks; and memory-based methods (e.g., [3, 6, 25]) aim to update the model with minimal interference introduced tooldtasks. More specifically, we first introduce notions of 'sufficient projection' and 'positive correlation' based on the gradient projection onto the subspaces of old tasks to characterize the task correlation.