Overcoming Catastrophic Forgetting in Convolutional Neural Networks by Selective Network Augmentation

Deep learning is a sub-field of machine learning which uses several learning algorithms to solve real-world tasks as image recognition, facial detection, signal processing, on supervised, unsupervised and reinforcement learning of feature representation at successively higher, more abstract layers. Those algorithms are artificial models such as Convolution Neural Networks (CNN), Deep Belief Networks (DBNs), Recurrent Neural Networks (RNNs) and Auto-encoders (AE). Even with the growth and success on many application of deep learning, some issues still remain unsolved in general. One of these issues is the catastrophic forgetting problem [1]. This issue can be seen as an handicap to develop truly intelligent systems. Catastrophic forgetting arises when a neural network is not capable of preserving the past learned task when learning new task. There are some approaches that benefit from previously learned information to improve performance of learning new information, for example fine-tuning [2] where the parameters of the old task are adjusted for adapting to a new task. Other approach well known is feature extraction [3] where the parameters of the old network are unchanged and the parameters of the outputs of one or more layers are used to extract feature for the new task.