Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks

Monte Carlo methods provide one solution to represent neural network parameter posteriors as ensembles of networks, but this requires In this paper, we present a general framework large amounts of both storage and compute time (Neal, for distilling expectations with respect to the 1996; Welling and Teh, 2011). Bayesian posterior distribution of a deep neural network classifier, extending prior work on To help overcome these problems, Balan et al. (2015) introduced the Bayesian Dark Knowledge framework. The a model training method referred to as Bayesian proposed framework takes as input "teacher" Dark Knowledge (BDK). BDK attempts to compress (or and student model architectures and a general distill) the Bayesian posterior predictive distribution induced posterior expectation of interest. The distillation by the full parameter posterior of a "teacher" network method performs an online compression (represented via a set of Mote Carlo samples) into a of the selected posterior expectation using iteratively significantly more compact "student" network. The major generated Monte Carlo samples. We advantage of BDK is that the computational complexity focus on the posterior predictive distribution of prediction at test time is drastically reduced compared and expected entropy as distillation targets. We to directly computing predictions via Monte Carlo averages investigate several aspects of this framework over the set of teacher network samples (the teacher including the impact of uncertainty and the ensemble).