Frosting Weights for Better Continual Training

--Training a neural network model can be a lifelong learning process and is a computationally intensive one. A severe adverse effect that may occur in deep neural network models is that they can suffer from catastrophic forgetting during retraining on new data. T o avoid such disruptions in the continuous learning, one appealing property is the additive nature of ensemble models. In this paper, we propose two generic ensemble approaches, gradient boosting and meta-learning, to solve the catastrophic forgetting problem in tuning pre-trained neural network models. With stationary training resources and various advanced neural network structures, deep learning models have exceeded human performance in many areas. However, a well-known limitation of deep learning models is the so-called "catastrophic forgetting."