We introduce a dynamic mechanism for the solution of analytically-tractable substructure in probabilistic programs, using conjugate priors and affine transformations to reduce variance in Monte Carlo estimators. For inference with Sequential Monte Carlo, this automatically yields improvements such as locally-optimal proposals and Rao-Blackwellization. The mechanism maintains a directed graph alongside the running program that evolves dynamically as operations are triggered upon it. Nodes of the graph represent random variables, edges the analytically-tractable relationships between them. Random variables remain in the graph for as long as possible, to be sampled only when they are used by the program in a way that cannot be resolved analytically. In the meantime, they are conditioned on as many observations as possible. We demonstrate the mechanism with a few pedagogical examples, as well as a linear-nonlinear state-space model with simulated data, and an epidemiological model with real data of a dengue outbreak in Micronesia. In all cases one or more variables are automatically marginalized out to significantly reduce variance in estimates of the marginal likelihood, in the final case facilitating a random-weight or pseudo-marginal-type importance sampler for parameter estimation. We have implemented the approach in Anglican and a new probabilistic programming language called Birch.

There has recently been a steadily increase in the iterative approaches to boosted density estimation and sampling, usually proceeding by adding candidate "iterate" densities to a model that gets more accurate with iterations. The relative accompanying burst of formal convergence results has not yet changed a striking picture: all results essentially pay the price of heavy assumptions on iterates, often unrealistic or hard to check, and offer a blatant contrast with the original boosting theory where such assumptions would be the weakest possible. In this paper, we show that all that suffices to achieve boosting for \textit{density estimation} is a \emph{weak learner} in the original boosting theory sense, that is, an oracle that supplies \textit{classifiers}. We provide converge rates that comply with boosting requirements, being better and / or relying on substantially weaker assumptions than the state of the art. One of our rates is to our knowledge the first to rely on not just weak but also \textit{empirically testable} assumptions. We show that the model fit belongs to exponential families, and obtain in the course of our results a variational characterization of $f$-divergences better than $f$-GAN's. Experimental results on several simulated problems display significantly better results than AdaGAN during early boosting rounds, in particular for mode capture, and using architectures less than the fifth's of AdaGAN's size.

In a recent paper, the authors proposed a general methodology for probabilistic learning on manifolds. The method was used to generate numerical samples that are statistically consistent with an existing dataset construed as a realization from a non-Gaussian random vector. The manifold structure is learned using diffusion manifolds and the statistical sample generation is accomplished using a projected Ito stochastic differential equation. This probabilistic learning approach has been extended to polynomial chaos representation of databases on manifolds and to probabilistic nonconvex constrained optimization with a fixed budget of function evaluations. The methodology introduces an isotropic-diffusion kernel with hyperparameter {\epsilon}. Currently, {\epsilon} is more or less arbitrarily chosen. In this paper, we propose a selection criterion for identifying an optimal value of {\epsilon}, based on a maximum entropy argument. The result is a comprehensive, closed, probabilistic model for characterizing data sets with hidden constraints. This entropy argument ensures that out of all possible models, this is the one that is the most uncertain beyond any specified constraints, which is selected. Applications are presented for several databases.

The Madrid ASDM summer school is in its thirteenth edition this year, with hundreds of students from all over the world having attended so far. It comprises 12 intensive (15 lecture hours) week-long courses, and a student may attend from one up to six courses. The courses cover topics such as Neural Networks and Deep Learning, Bayesian Networks, Big Data with Apache Spark, Bayesian Inference, Text Mining and Time Series. Each course has theoretical and practical classes, the latter done with R or python. While the summer school is mainly attended by people from academia - PhD students and researchers-, people from the industry also assist.

Probabilistic graphical models are a key tool in machine learning applications. Computing the partition function, i.e., normalizing constant, is a fundamental task of statistical inference but it is generally computationally intractable, leading to extensive study of approximation methods. Iterative variational methods are a popular and successful family of approaches. However, even state of the art variational methods can return poor results or fail to converge on difficult instances. In this paper, we instead consider computing the partition function via sequential summation over variables. We develop robust approximate algorithms by combining ideas from mini-bucket elimination with tensor network and renormalization group methods from statistical physics. The resulting "convergence-free" methods show good empirical performance on both synthetic and real-world benchmark models, even for difficult instances.

In this work, we investigate Batch Normalization technique and propose its probabilistic interpretation. We propose a probabilistic model and show that Batch Normalization maximazes the lower bound of its marginalized log-likelihood. Then, according to the new probabilistic model, we design an algorithm which acts consistently during train and test. However, inference becomes computationally inefficient. To reduce memory and computational cost, we propose Stochastic Batch Normalization -- an efficient approximation of proper inference procedure. This method provides us with a scalable uncertainty estimation technique. We demonstrate the performance of Stochastic Batch Normalization on popular architectures (including deep convolutional architectures: VGG-like and ResNets) for MNIST and CIFAR-10 datasets.

The central aim in this paper is to address variable selection questions in nonlinear and nonparametric regression. Motivated by statistical genetics, where nonlinear interactions are of particular interest, we introduce a novel, interpretable, and computationally efficient way to summarize the relative importance of predictor variables. Methodologically, we develop the "RelATive cEntrality" (RATE) measure to prioritize candidate genetic variants that are not just marginally important, but whose associations also stem from significant covarying relationships with other variants in the data. We illustrate RATE through Bayesian Gaussian process regression, but the methodological innovations apply to other nonlinear methods. It is known that nonlinear models often exhibit greater predictive accuracy than linear models, particularly for phenotypes generated by complex genetic architectures. With detailed simulations and an Arabidopsis thaliana QTL mapping study, we show that applying RATE enables an explanation for this improved performance.

This paper introduces a nonparametric copula-based index for detecting the strength and monotonicity structure of linear and nonlinear statistical dependence between pairs of random variables or stochastic signals. Our index, termed Copula Index for Detecting Dependence and Monotonicity (CIM), satisfies several desirable properties of measures of association, including R\'enyi's properties, the data processing inequality (DPI), and consequently self-equitability. Synthetic data simulations reveal that the statistical power of CIM compares favorably to other state-of-the-art measures of association that are proven to satisfy the DPI. Simulation results with real-world data reveal the CIM's unique ability to detect the monotonicity structure among stochastic signals to find interesting dependencies in large datasets. Additionally, simulations show that the CIM shows favorable performance to estimators of mutual information when discovering Markov network structure.

A new Bayesian modeling method is proposed by combining the maximization of the marginal likelihood with a momentum-space renormalization group transformation for Gaussian graphical models. Moreover, we present a scheme for computint the statistical averages of hyperparameters and mean square errors in our proposed method based on a momentumspace renormalization transformation.

A Gaussian process (GP) [1] is a prior distribution over functions with a support that includes a wide class of phenomena via the design of its mean and covariance functions, the parameters of which provide meaningful interpretation of the process at hand. Beyond regression [2], GPs have been extensively used in the last two decades for classification [3], density estimation [4], filter design [5], model identification [6] and optimisation [7]. In general terms, all these generative models have two stages: The latent process is modelled as a GP and the observation is modelled (conditional to the latent process) as a non-Gaussian variable. This class of models is referred to as GP with non-Gaussian likelihood, or as Generalised GPs. These usually consider likelihood functions from the exponential family such as the Laplace, Poisson, beta and gamma distributions [8]. A well-known example is the GP classification model, where the classes are represented by the output of an activation neuron into which a latent GP is fed. A slightly different approach to non-Gaussian models, which is not constrained to the exponential family, is the warped GP (WGP, [9]). The WGP models non-Gaussian data by assuming that there is a transformation φ such that the observations can be passed through φ to yield a GP, therefore, the likelihood function of this model is not designed directly but, rather, induced by the transformation (a.k.a.