Regularization and Learning an Ensemble of RNNs by Decorrelating Representations

Recurrent Neural Networks (RNNs) and their variants (suchas LSTMs and GRUs) have been remarkably successful atmachine-learning tasks on diverse kinds of sequential data(e.g. text, time-series, etc.). However, training of RNNs con-tinue to be a challenge due to difficulties stemming from regu-larization and the highly non-convex optimizations involved.In this paper, we propose to regularize training of RNNs byencouraging higher decorrelation in the hidden representa-tions. The cost function is devised to minimize non-diagonalelements of the correlation matrix computed over the hid-den representations of RNNs, along with the usual trainingaccuracy term; thereby penalizing redundancy in the learnedmodel. Furthermore, we propose to utilize the idea of decor-relating representations in learning an ensemble of RNNs,in order to maximize diversity in the resulting models; thusenforcing every individual network of the ensemble to gainabilities that are complementary to the ensemble. Extensiveexperiments are presented on various datasets with differentarchitectures of RNNs. Results are offered for multiple tasksand show that the proposed methods yield a significant im-provement; when compared with the state-of-the-art methods.