Stochastic Neural Networks with Monotonic Activation Functions

Siamak Ravanbakhsh, Barnab as P oczos, Jeff Schneider 1 and Dale Schuurmans, Russell Greiner 2 1 Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA 15213 2 University of Alberta, Edmonton, AB T6G 2E8, Canada Abstract We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences. This family, that we call exponential family RBM (Exp-RBM), is a subset of the exponential family Harmoniums that expresses family members through a choice of smooth monotonic non-linearity for each neuron. Using contrastive divergence along with our Gaussian approximation, we show that Exp-RBM can learn useful representations using novel stochastic units. 1 Introduction Deep neural networks (LeCun et al., 2015; Bengio, 2009) have produced some of the best results in complex pattern recognition tasks where the training data is abundant. Here, we are interested in deep learning for generative modeling. Recent years has witnessed a surge of interest in directed generative models that are trained using (stochastic) back-propagation ( e.g., Kingma and Welling, 2013; Rezende et al., 2014; Goodfellow et al., 2014). These models are distinct from deep energy-based models - including deep Boltzmann machine (Hinton et al., 2006) and (convolutional) deep belief networkAppearing in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS) 2016, Cadiz, Spain. Although, due to their use of Gaussian noise, the stochastic units that we introduce in this paper can be potentially used with stochastic back-propagation, this paper is limited to applications in RBM.