Entropy and mutual information in models of deep neural networks

We examine a class of stochastic deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are threefold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physicsmethods, under the assumption that weight matrices are independent and orthogonally-invariant. (ii) We extend particular cases in which this result is known to be rigorously exact by providing a proof for two-layers networks with Gaussian random weights, using the recently introduced adaptive interpolation method.