Goto

Collaborating Authors

Concrete Dropout

Neural Information Processing Systems

Dropout is used as a practical tool to obtain uncertainty estimates in large vision models and reinforcement learning (RL) tasks. But to obtain well-calibrated uncertainty estimates, a grid-search over the dropout probabilities is necessary--a prohibitive operation with large models, and an impossible one with RL. We propose a new dropout variant which gives improved performance and better calibrated uncertainties. Relying on recent developments in Bayesian deep learning, we use a continuous relaxation of dropout's discrete masks. Together with a principled optimisation objective, this allows for automatic tuning of the dropout probability in large models, and as a result faster experimentation cycles.


Simple and Accurate Uncertainty Quantification from Bias-Variance Decomposition

arXiv.org Machine Learning

Examples include medical diagnosis and selfdriving (Kennedy & O'Hagan, 2001) provides a more fine-grained vehicles. We propose a new method that categorization of uncertainty into six terms. Among them, is based directly on the bias-variance decomposition, the parameter and experimental uncertainties correspond where the parameter uncertainty is given by to the epistemic and aleatoric uncertainties in (Kendall & the variance of an ensemble divided by the number Gal, 2017), and the structural uncertainty corresponds to of members in the ensemble, and the aleatoric the missing model bias. For clarity, from now on we switch uncertainty plus the squared bias is estimated by to the uncertainty terminologies defined in (Kennedy & training a separate model that is regressed directly O'Hagan, 2001) for the rest of this paper.


Concrete Dropout

arXiv.org Machine Learning

Dropout is used as a practical tool to obtain uncertainty estimates in large vision models and reinforcement learning (RL) tasks. But to obtain well-calibrated uncertainty estimates, a grid-search over the dropout probabilities is necessary - a prohibitive operation with large models, and an impossible one with RL. We propose a new dropout variant which gives improved performance and better calibrated uncertainties. Relying on recent developments in Bayesian deep learning, we use a continuous relaxation of dropout's discrete masks. Together with a principled optimisation objective, this allows for automatic tuning of the dropout probability in large models, and as a result faster experimentation cycles. In RL this allows the agent to adapt its uncertainty dynamically as more data is observed. We analyse the proposed variant extensively on a range of tasks, and give insights into common practice in the field where larger dropout probabilities are often used in deeper model layers.


Engineering Uncertainty Estimation in Neural Networks for Time Series Prediction at Uber

@machinelearnbot

Accurate time series forecasting during high variance segments (e.g., holidays and sporting events) is critical for anomaly detection, resource allocation, budget planning, and other related tasks necessary to facilitate optimal Uber user experiences at scale. Forecasting these variables, however, can be challenging because extreme event prediction depends on weather, city population growth, and other external factors that contribute to forecast uncertainty. In recent years, the Long Short Term Memory (LSTM) technique has become a popular time series modeling framework due to its end-to-end modeling, ease of incorporating exogenous variables, and automatic feature extraction abilities. Uncertainty estimation in deep learning remains a less trodden but increasingly important component of assessing forecast prediction truth in LSTM models. Through our research, we found that a neural network forecasting model is able to outperform classical time series methods in use cases with long, interdependent time series. While beneficial in other ways, our new model did not offer insights into prediction uncertainty, which helps determine how much we can trust the forecast.


Spatial Uncertainty Sampling for End-to-End Control

arXiv.org Artificial Intelligence

End-to-end trained neural networks (NNs) are a compelling approach to autonomous vehicle control because of their ability to learn complex tasks without manual engineering of rule-based decisions. However, challenging road conditions, ambiguous navigation situations, and safety considerations require reliable uncertainty estimation for the eventual adoption of full-scale autonomous vehicles. Bayesian deep learning approaches provide a way to estimate uncertainty by approximating the posterior distribution of weights given a set of training data. Dropout training in deep NNs approximates Bayesian inference in a deep Gaussian process and can thus be used to estimate model uncertainty. In this paper, we propose a Bayesian NN for end-to-end control that estimates uncertainty by exploiting feature map correlation during training. This approach achieves improved model fits, as well as tighter uncertainty estimates, than traditional element-wise dropout. We evaluate our algorithms on a challenging dataset collected over many different road types, times of day, and weather conditions, and demonstrate how uncertainties can be used in conjunction with a human controller in a parallel autonomous setting.