We study the problem of off-policy evaluation (OPE) in Reinforcement Learning (RL), where the aim is to estimate the performance of a new policy given historical data that may have been generated by a different policy, or policies. In particular, we introduce a novel doubly-robust estimator for the OPE problem in RL, based on the Targeted Maximum Likelihood Estimation principle from the statistical causal inference literature. We also introduce several variance reduction techniques that lead to impressive performance gains in off-policy evaluation. We show empirically that our estimator uniformly wins over existing off-policy evaluation methods across multiple RL environments and various levels of model misspecification. Finally, we further the existing theoretical analysis of estimators for the RL off-policy estimation problem by showing their $O_P(1/\sqrt{n})$ rate of convergence and characterizing their asymptotic distribution.

Evidential clustering is an approach to clustering in which cluster-membership uncertainty is represented by a collection of Dempster-Shafer mass functions forming an evidential partition. In this paper, we propose to construct these mass functions by bootstrapping finite mixture models. In the first step, we compute bootstrap percentile confidence intervals for all pairwise probabilities (the probabilities for any two objects to belong to the same class). We then construct an evidential partition such that the pairwise belief and plausibility degrees approximate the bounds of the confidence intervals. This evidential partition is calibrated, in the sense that the pairwise belief-plausibility intervals contain the true probabilities "most of the time", i.e., with a probability close to the defined confidence level. This frequentist property is verified by simulation, and the practical applicability of the method is demonstrated using several real datasets.

Department of Physics, Department of Astronomy University of California, Berkeley, CA 94720, USA and Lawrence Berkeley National Lab, 1 Cyclotron Road, Berkeley, CA 94720, USA Abstract Normalizing constant (also called partition function, Bayesian evidence, or marginal likelihood) is one of the central goals of Bayesian inference, yet most of the existing methods are both expensive and inaccurate. Here we develop a new approach, starting from posterior samples obtained with a standard Markov Chain Monte Carlo (MCMC). We apply a novel Normalizing Flow (NF) approach to obtain an analytic density estimator from these samples, followed by Optimal Bridge Sampling (OBS) to obtain the normalizing constant. We compare our method which we call Gaussianized Bridge Sampling (GBS) to existing methods such as Nested Sampling (NS) and Annealed Importance Sampling (AIS) on several examples, showing our method is both significantly faster and substantially more accurate than these methods, and comes with a reliable error estimation. Keywords: Normalizing Constant, Bridge Sampling, Normalizing Flows 1. Introduction Normalizing constant, also called partition function, Bayesian evidence, or marginal likelihood, is the central object of Bayesian methodology.

Model misspecification is a long-standing enigma of the Bayesian inference framework as posteriors tend to get overly concentrated on ill-informed parameter values towards the large sample limit. Tempering of the likelihood has been established as a safer way to do updates from prior to posterior in the presence of model misspecification. At one extreme tempering can ignore the data altogether and at the other extreme it provides the standard Bayes' update when no misspecification is assumed to be present. However, it is an open issue how to best recognize misspecification and choose a suitable level of tempering without access to the true generating model. Here we show how probabilistic classifiers can be employed to resolve this issue. By training a probabilistic classifier to discriminate between simulated and observed data provides an estimate of the ratio between the model likelihood and the likelihood of the data under the unobserved true generative process, within the discriminatory abilities of the classifier. The expectation of the logarithm of a ratio with respect to the data generating process gives an estimation of the negative Kullback-Leibler divergence between the statistical generative model and the true generative distribution. Using a set of canonical examples we show that this divergence provides a useful misspecification diagnostic, a model comparison tool, and a method to inform a generalised Bayesian update in the presence of misspecification for likelihood-based models.

While Bayesian neural networks have many appealing characteristics, current priors do not easily allow users to specify basic properties such as expected lengthscale or amplitude variance. In this work, we introduce Poisson Process Radial Basis Function Networks, a novel prior that is able to encode amplitude stationarity and input-dependent lengthscale. We prove that our novel formulation allows for a decoupled specification of these properties, and that the estimated regression function is consistent as the number of observations tends to infinity. We demonstrate its behavior on synthetic and real examples.

Research has found that many occupational accidents are foreseeable, being the result of people's unsafe behaviour from a retrospective point of view. The prediction of workers' safety behaviour will enable the prior insights into each worker's behavioural tendency and will be useful in the design of management practices prior to the occurrence of accidents and contribute to the reduction of injury rates. In recent years, researchers have found that people do have stable predispositions to engage in certain safety behavioural patterns which vary among individuals as a function of personality features. In this study, an innovative forecasting model, which employs machine learning algorithms, is developed to estimate construction workers' behavioural tendency based on the Big Five personality taxonomy. The data-driven nature of machine learning technique enabled a reliable estimate of the personality-safety behaviour relationship, which allowed this study to provide novel insight that nonlinearity may exist in the relationship between construction workers' personality traits and safety behaviour. The developed model is found to be sufficient to have satisfactory accuracy in explaining and predicting workers' safety behaviour. This finding provides the empirical evidence to support the usefulness of personality traits as effective predictors of people's safety behaviour at work. In addition, this study could have practical implications. The machine learning model developed can help identify vulnerable workers who are more prone to undertake unsafe behaviours, which is proven to have good prediction accuracy and is thereby potentially useful for decision making and safety management on construction sites.

In these online tools, patients input their initial symptoms and then proceed to answer a series of questions that the system deems relevant to those symptoms. The output of these online tools is a differential diagnosis (ranked list of diseases) that helps educate patients on possible relevant health conditions. Online symptom checkers are powered by underlying diagnosis models or engines similar to those used for advising physicians in "clinical decision support tools"; the main difference in this scenario being that the resulting differential diagnosis is not directly shared with the patient, but rather used by a physician for professional evaluation. Diagnosis models must have high accuracy while covering a large space of symptoms and diseases to be useful to patients and physicians. Accuracy is critically important, as incorrect diagnoses can give patients unnecessary cause for concern.

Motivation: Untargeted metabolomics comprehensively characterizes small molecules and elucidates activities of biochemical pathways within a biological sample. Despite computational advances, interpreting collected measurements and determining their biological role remains a challenge. Results: To interpret measurements, we present an inference-based approach, termed Probabilistic modeling for Untargeted Metabolomics Analysis (PUMA). Our approach captures measurements and known information about the sample under study in a generative model and uses stochastic sampling to compute posterior probability distributions. PUMA predicts the likelihood of pathways being active, and then derives a probabilistic annotation, which assigns chemical identities to the measurements. PUMA is validated on synthetic datasets. When applied to test cases, the resulting pathway activities are biologically meaningful and distinctly different from those obtained using statistical pathway enrichment techniques. Annotation results are in agreement to those obtained using other tools that utilize additional information in the form of spectral signatures. Importantly, PUMA annotates many additional measurements.

Multitask learning (MTL) is a learning paradigm in which multiple tasks are learned jointly, aiming to improve the performance of individual tasks by sharing information across tasks [4, 26], using various information sharing mechanisms. For example, MTL models based on deep neural networks commonly use shared hidden layers for all the tasks; probabilistic MTL models are usually based on shared priors over the parameters of the multiple tasks [16, 5]; Gaussian Process based models, e.g., multitask Gaussian Processes (GP) and extensions [2, 23], commonly employ covariance functions that models both inputs and task similarity. Multi-label, multi-class, multi-output learning can be seen as special cases of multitask learning where each task has the same set of inputs. Transfer learning is also similar to MTL, except that the objective of MTL is to improve the performance over all the tasks whereas the objective of transfer learning is to usually improve the performance of a target task by leveraging information from source tasks [26]. Zero-shot learning and few-shot learning are also closely related to MTL. Prior works [14, 24] have shown that a fully connected Bayesian neural network (NN) [13, 15] with a single, infinitely-wide hidden layer, with independent and identically distributed (i.i.d) priors on weights, is equivalent to a Gaussian Process. The result has recently been also generalized to deep Bayesian neural networks [9] with any number of hidden layers. These connections between Bayesian neural networks and GP offer many benefits, such as theoretical understanding of neural networks, efficient Bayesian inference for deep NN by learning the equivalent GP, etc. Motivated by the equivalence of deep Bayesian neural networks and GP, in this work, we investigate whether a similar connection exists between deep multitask Bayesian neural networks [18] and multitask Gaussian Processes

As the size and richness of available datasets grow larger, the opportunities for solving increasingly challenging problems with algorithms learning directly from data grow at the same pace. Consequently, the capability of learning algorithms to work with large amounts of data has become a crucial scientific and technological challenge for their practical applicability. Hence, it is no surprise that large-scale learning is currently drawing plenty of research effort in the machine learning research community. In this thesis, we focus on kernel methods, a theoretically sound and effective class of learning algorithms yielding nonparametric estimators. Kernel methods, in their classical formulations, are accurate and efficient on datasets of limited size, but do not scale up in a cost-effective manner. Recent research has shown that approximate learning algorithms, for instance random subsampling methods like Nystr\"om and random features, with time-memory-accuracy trade-off mechanisms are more scalable alternatives. In this thesis, we provide analyses of the generalization properties and computational requirements of several types of such approximation schemes. In particular, we expose the tight relationship between statistics and computations, with the goal of tailoring the accuracy of the learning process to the available computational resources. Our results are supported by experimental evidence on large-scale datasets and numerical simulations. We also study how large-scale learning can be applied to enable accurate, efficient, and reactive lifelong learning for robotics. In particular, we propose algorithms allowing robots to learn continuously from experience and adapt to changes in their operational environment. The proposed methods are validated on the iCub humanoid robot in addition to other benchmarks.