Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning

Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning. Extending a dataset with new samples to train a deep learning model typically poses two problems. Updating a trained model may cause catastrophic forgetting while retraining may require high computational effort. Although the generalization performance typically justifies exhaustive retraining procedures, in some applications, retraining is not possible due to, for example, (1) the high number of retraining procedures in applications where data arrives sequentially and immediate updates are beneficial, e.g., in active learning (Settles, 2009) or when working on data streams (Sahoo et al., 2018), (2) the lack of computational power, e.g., for execution on embedded hardware (Taylor et al., 2018), (3) privacy reasons, e.g., when new data cannot be sent to distributed computing units (Taylor et al., 2018). Therefore, we suggest using Bayesian neural networks (BNNs, Fortuin, 2022) in the above examples as they not only provide additional uncertainty estimates or out-of-distribution detection capabilities but also allow updating the predictions with additional data without retraining the network (Kirsch et al., 2022). In this article, we develop a fast Bayesian update algorithm for BNNs. Figure 1 (left) shows the idea of sampling an ensemble of probabilistic hypotheses, each representing a possible true solution for the learning task (white samples).