Review for NeurIPS paper: Bayesian Deep Learning and a Probabilistic Perspective of Generalization

After much discussion, the reviewers largely converged towards recommending to accept this submission. The reviewers appreciate the merits of the paper, believe it investigates important open questions, and will thus be a significant contribution to our understanding of BNNs, but only when the experimental issues mentioned in the reviews are resolved. I would draw the author's attention to the fact that the reviewers raised concerns about the supplementary material containing a number of sections which are not connected to results in the main paper (on tempered posteriors, sampling from the prior, discussions of what's Bayesian, PAC Bayes etc.). Per reviewing guidelines, since these sections were not relevant for understanding the main paper, these were not reviewed with scrutiny. However, the reviewers found strong statements in the unreviewed supplementary material involving other recent work which they believe deserve close scrutiny if they are to be published.