Statistical Inference is the process of determining a probability distribution over the space of parameters of a model given a data set. As more data becomes available this probability distribution becomes updated via the application of Bayes' theorem. We present a treatment of this Bayesian updating process as a continuous dynamical system. Statistical inference is then governed by a first order differential equation describing a trajectory or flow in the information geometry determined by a parametric family of models. We solve this equation for some simple models and show that when the Cram\'{e}r-Rao bound is saturated the learning rate is governed by a simple $1/T$ power-law, with $T$ a time-like variable denoting the quantity of data. The presence of hidden variables can be incorporated in this setting, leading to an additional driving term in the resulting flow equation. We illustrate this with both analytic and numerical examples based on Gaussians and Gaussian Random Processes and inference of the coupling constant in the 1D Ising model. Finally we compare the qualitative behaviour exhibited by Bayesian flows to the training of various neural networks on benchmarked data sets such as MNIST and CIFAR10 and show how that for networks exhibiting small final losses the simple power-law is also satisfied.

Stein variational gradient descent (SVGD) is a general-purpose optimization-based sampling algorithm that has recently exploded in popularity, but is limited by two issues: it is known to produce biased samples, and it can be slow to converge on complicated distributions. A recently proposed stochastic variant of SVGD (sSVGD) addresses the first issue, producing unbiased samples by incorporating a special noise into the SVGD dynamics such that asymptotic convergence is guaranteed. Meanwhile, Stein variational Newton (SVN), a Newton-like extension of SVGD, dramatically accelerates the convergence of SVGD by incorporating Hessian information into the dynamics, but also produces biased samples. In this paper we derive, and provide a practical implementation of, a stochastic variant of SVN (sSVN) which is both asymptotically correct and converges rapidly. We demonstrate the effectiveness of our algorithm on a difficult class of test problems -- the Hybrid Rosenbrock density -- and show that sSVN converges using three orders of magnitude fewer gradient evaluations of the log likelihood than its stochastic SVGD counterpart. Our results show that sSVN is a promising approach to accelerating high-precision Bayesian inference tasks with modest-dimension, $d\sim\mathcal{O}(10)$.

The Bayesian information criterion (BIC), defined as the observed data log likelihood minus a penalty term based on the sample size $N$, is a popular model selection criterion for factor analysis with complete data. This definition has also been suggested for incomplete data. However, the penalty term based on the `complete' sample size $N$ is the same no matter whether in a complete or incomplete data case. For incomplete data, there are often only $N_i

In unsupervised learning, Bayesian Nonparametric (BNP) mixture models, exemplified by the Dirichlet-Process Mixture Model (DPMM), provide a principled approach for Bayesian modeling while adapting the model complexity to the data. This contrasts with finite mixture models whose complexity is determined manually or via model-selection methods. To fix ideas, an important DPMM example is the Dirichlet-Process Gaussian Mixture Model (DPGMM), a Bayesian -dimensional extension of the classical Gaussian Mixture Model (GMM). Despite their potential, however, and although researchers have used them successfully in numerous applications during the last two decades, DPMMs still do not enjoy wide popularity among practitioners, largely due to computational bottlenecks that exist in current algorithms and/or implementations. In particular, one of the missing pieces is the availability of software tools that: 1) can efficiently handle DPMM inference in large datasets; 2) are user-friendly and can also be easily modified. We argue that in order for DPMMs to become a practical choice for large-scale data analysis, implementations of DPMM inference must leverage parallel-and distributed-computing resources (in an analogy, consider how advances in GPU computing and GPU software contributed to the success of deep learning). This is because of not only potential speedups but also memory and storage considerations. For example, this is especially true in distributed mobile robotic sensing applications where multiple autonomous agents working together have limited computational and communication resources. As another motivating example, consider unsupervised dataanalysis tasks in large and high-dimensional computer-vision datasets.

As a powerful statistical tool that has revolutionized modern molecular evolutionary analysis, Bayesian phylogenetic inference has been widely used for tasks ranging from genomic epidemiology [Dudas et al., 2017, du Plessis et al., 2021] to conservation genetics [DeSalle and Amato, 2004]. Given aligned sequence data (e.g., DNA, RNA or protein sequences) and a model of evolution, Bayesian phylogenetics provides principled approaches to quantify the uncertainty of the evolutionary process in terms of the posterior probabilities of phylogenetic trees [Huelsenbeck et al., 2001]. In addition to uncertainty quantification, Bayesian methods enable integrating out tree uncertainty in order to get more confident estimates of parameters of interest, such as factors in the transmission of Ebolavirus [Dudas et al., 2017]. Bayesian methods also allow complex substitution models [Lartillot and Philippe, 2004], which are important in elucidating deep phylogenetic relationships [Feuda et al., 2017]. Ever since its introduction to the phylogenetic community in the 1990s, Bayesian phylogenetic inference has been dominated by random-walk Markov chain Monte Carlo (MCMC) approaches [Yang and Rannala, 1997, Mau et al., 1999, Huelsenbeck and Ronquist, 2001, Drummond et al., 2002, 2005]. However, this approach is fundamentally limited by the complexities of tree space.

Photoacoustic tomography (PAT) is a hybrid biomedical imaging modality based on the photoacoustic effect [6, 44, 32]. In PAT, the imaged target is illuminated with a short pulse of light. Absorption of light creates localized areas of thermal expansion, resulting in localized pressure increases within the imaged target. This pressure distribution, called the initial pressure, relaxes as broadband ultrasound waves that are measured on the boundary of the imaged target. In the inverse problem of PAT, the initial pressure distribution is estimated from a set of measured ultrasound data.

Probabilistic programming is a growing area that strives to make statistical analysis more accessible, by separating probabilistic modelling from probabilistic inference. In practice this decoupling is difficult. No single inference algorithm can be used as a probabilistic programming back-end that is simultaneously reliable, efficient, black-box, and general. Probabilistic programming languages often choose a single algorithm to apply to a given problem, thus inheriting its limitations. While substantial work has been done both to formalise probabilistic programming and to improve efficiency of inference, there has been little work that makes use of the available program structure, by formally analysing it, to better utilise the underlying inference algorithm. This dissertation presents three novel techniques (both static and dynamic), which aim to improve probabilistic programming using program analysis. The techniques analyse a probabilistic program and adapt it to make inference more efficient, sometimes in a way that would have been tedious or impossible to do by hand.

To leave a comment for the author, please follow the link and comment on their blog: K & L Fintech Modeling. R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job. Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. To leave a comment for the author, please follow the link and comment on their blog: K & L Fintech Modeling.

Machine learning (ML) has become ubiquitous in scientific disciplines. In some applications, accurate data-driven predictions are all that is required; however, in many others, interpretability and explainability of the model is equally important. Interpretability and explainability can provide justification for decisions, promote scientific discovery and ultimately lead to better control/improvement of models [1, 2]. In a complementary fashion, ML models can provide further insight by conveying their level of uncertainty in predictions [3]. Especially in cases of low risk tolerance this type of insight is crucial for building trust in ML models [4]. Rather than focus on black-box ML methods (e.g., neural networks or Gaussian process regression) combined with post hoc explainability tools, the current work focuses on inherently interpretable methods. Interpretable ML methods can be competitive with black-box ML in terms of accuracy and do not require a separate explainability toolkit [4, 5]. Symbolic regression is one such inherently interpretable form of ML wherein an analytic equation is produced that best models input data.

In the road so far, we have talked about MLP, CNN, and RNN architectures. These are discriminative models, that is models that can make predictions. Discriminative models essentially learn to estimate a conditional probability distribution p( x); that is, given a value, they try to predict the outcome based on what they learned about the probability distribution of x. Generative models are architectures of neural networks that learn the probability distribution of the data and learn how to generate data that seems to come from that probability distribution. Creating synthetic data is one use of generative models, but is not the only one.