Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks

Mean field theory has been successfully used to analyze deep neura l networks (DNN) in the infinite size limit. Given the finite size of realistic D NN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solution s. The parameter perturbations investigated include weight sparsification (dilution) a nd binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are m ore robust to parameter perturbations with respect to their counterparts wit h sign activation, which arguably is reflected in the simplicity of the functions they generate . Keywords: large deviation theory, path integral, deep neural networks, fu nction sensitivity 1. Introduction Learning machines realized by deep neural networks (DNN) have ac hieved impressive success in performing various machine learning tasks, such as spee ch recognition, image classification and natural language processing [1].