How good is PAC-Bayes at explaining generalisation?

The widespread use of modern neural networks for high-stakes applications requires safety guarantees on their performance on future data, which have not been observed during the training [Xu and Goodacre, 2018, Russell and Norvig, 2020]. A well-established approach to train and evaluate the performance of a predictor consists of the following steps. First, the available data are split in a train and a test datasets. The training data are used to construct the predictor, whose performance is then assessed on the test data (empirical test risk). Finally, concentration inequalities [Boucheron et al., 2013] are used to derive, from this finite-sample test, an upper bound on the model expected performance over the data distribution (population risk) [Langford, 2005].