Reviews: Natural-Parameter Networks: A Class of Probabilistic Neural Networks

The paper presents a novel and potentially impactful way of learning uncertainty over model parameters. The derivation of novel activation functions for which first and second moments are computable in closed forms (for distributions in the exponential family) appears to be the main (novel) contribution, as this is what allows forward propagation of exponential distributions in the network, and learning of their parameters via backprop. The work does bear some resemblance to earlier work on "Implicit Variance Networks" Bayer et al. which ought to be discussed. On a technical level, the method appears to be effective and the authors empirically verify that: (1) the method is robust to overfitting (2) predictive uncertainty is well calibrated and (3) that propagating distributions over latent states can outperform deterministic methods (e.g. The fact that these second order representations outperform those of VAE is somewhat more surprising and may warrants further experimentation: this would imply that the approximation used by the VAE at inference, is worse than the approximation made by NPN that each layer's activation belongs to the exponential family.