Regularising Deep Networks with DGMs

Regularising Deep Networks with DGMsMatthew Willetts, Alexander Camuto, Stephen Roberts, Chris Holmes University of Oxford Alan Turing Institute {mwilletts, acamuto, sroberts, cholmes }@turing.ac.uk Abstract Here we develop a new method for regularising neural networks where we learn a density estimator over the activations of all layers of the model. We extend recent work in data imputation using V AEs (Ivanov et al., 2018) so that we can obtain a posterior for an arbitrary subset of activations conditioned on the remainder. Our method has links both to dropout and to data augmentation. We demonstrate that our training method leads to lower cross-entropy test set loss for 2-hidden-layer neural networks trained on CIFAR-10 and SVHN compared to standard regularisation baselines, but our model does not improve test-set accuracy compared to our baselines. This implies that although decisions are broadly similar, our approach provides a network with better calibrated uncertainty measures over the class posteriors. 1 Introduction Methods such a dropout [1], batch norm [2], L 2 regularisation [3] and data augmentation [3, 4] improve generalisation and robustness of deep discriminative models p Ψ(y x).