Reviews: Bayesian optimization for automated model selection

I think the clarity of your presentation can be strengthened. Sure, I'd like to integrate out the hyperparameters (and thus average over a class of models), but if I can't do that, then I'm happy to learn them and select a single model. Indeed, I'd question your claim that the Bayesian perspective is one of Bayesian model selection---you're just moving one more level up the hierarchy (hyper-hyper parameters!), but wouldn't it be even better if we could integrate over different models M? This might actually be possible within your framework---you're using Bayesian optimization to maximize a function, but the recent probabilistic numerics work on Bayesian quadrature suggests you could instead use BO to do integration! I think run an optimization to approximate the model evidence, is that right? If so, it's confusing me because the model evidence approximation isn't discussed until later -- page 5 while the acquisition function is discussed on page 3. 6) The priors you suggest are reasonable, though I wonder if it'd be better to use a distribution like Student t (see some work by Vehtari and colleagues) in place of a Normal.