On Model Mismatch and Bayesian Analysis

One aspect I always enjoy about machine learning is that questions often go back to the basics. The field essentially goes into an existential crisis every dozen years--rethinking our tools and asking foundational questions such as "why neural networks" or "why generative models".1 This was a theme in my conversations during NIPS 2016 last week, where a frequent topic was on the advantages of a Bayesian perspective to machine learning. Not surprisingly, this appeared as a big discussion point during the panel at the Bayesian deep learning workshop, where many panelists were conciliatory to the use of non-Bayesian approaches. While Bayesian inference can capture uncertainty about parameters, it relies on the model being correctly specified.