Target Layer Regularization for Continual Learning Using Cramer-Wold Generator

The concept of continual learning (CL), which aims to reduce the distance between human and artificial intelligence, seems to be considered recently by deep learning community as one of the main challenges. Generally speaking, it means the ability of the neural network to effectively learn consecutive tasks (in either supervised or unsupervised scenarios) while trying to prevent forgetting already learned information. Therefore, when designing an appropriate strategy, it needs to be ensured that the network weights are updated in such a way that they correspond to both the current and all previous tasks. However, in practice, it is quite likely that constructed CL model will suffer from either intransigence (hard acquiring new knowledge, see Chaudhry et al. [2018]) or catastrophic forgetting (CF) phenomenon (tendency to lose past knowledge, see McCloskey and Cohen [1989]). In recent years, methods of overcoming the above-mentioned problems are subject to wide and intensive investigation.