Regularizing Neural Networks by Stochastically Training Layer Ensembles

REGULARIZING NEURAL NETWORKS BY STOCHASTICALL Y TRAINING LA YER ENSEMBLES Alex Labach and Shahrokh V alaee University of Toronto Department of Electrical and Computer Engineering Toronto, Canada ABSTRACT Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic regularization while explicitly averaging outputs. This provides stronger regularization with no additional computational cost at test time. We show consistent improvement on various image classification tasks using standard network topologies. Index T erms-- neural networks, regularization, dropout, model averaging, ensemble methods 1. INTRODUCTION In order to generalize well to new inputs, modern deep neural networks require heavy regularization.