Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.

The paper investigates the use of an ensemble of neural networks (each modelling the probability of the target given the input and trained with adversarial training) for quantifying predictive uncertainty. A series of experiments shows that this simple approach archives competitive results to the standard Bayesian models. While the proposed approach is based on well-known models and techniques (and thus is not new itself), to the best of my knowledge it has not been applied to the problem of estimating predictive uncertainty so far and could serve as a good benchmark in future. A drawback compared to the Bayesian models is that the approach comes without mathematical framework and guarantees. Specific comments and questions: - While other approaches, like MC-dropout can also be applied to regression problems with multidimensional targets, it is not clear to me, if a training criterion (like that described in section 2.2.1) suitable for multidimensional targets does also exists (learning a multivariate Gaussian with dependent variables seems not state forward).

Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs.