A Bayesian Inference over Neural Networks

The prior and likelihood are both modelling choices. Since (14) is intractable, we typically sample a finite set of parameters and compute a Monte Carlo estimator. A.1 Likelihoods for BNNs The likelihood is purely a function of the model prediction Φ As such, BNN likelihood distributions follow the standard choices used in other probabilistic models. Neal [21] shows that in the regression setting, the isotropic Gaussian prior for a BNN with a single hidden layer approaches a Gaussian process prior as the number of hidden units tends to infinity, so long as the chosen activation function is bounded. We will use this prior in the baseline BNN for our experiments.