We introduce a novel geometry-informed irreversible perturbation that accelerates convergence of the Langevin algorithm for Bayesian computation. It is well documented that there exist perturbations to the Langevin dynamics that preserve its invariant measure while accelerating its convergence. Irreversible perturbations and reversible perturbations (such as Riemannian manifold Langevin dynamics (RMLD)) have separately been shown to improve the performance of Langevin samplers. We consider these two perturbations simultaneously by presenting a novel form of irreversible perturbation for RMLD that is informed by the underlying geometry. Through numerical examples, we show that this new irreversible perturbation can improve performance of the estimator over reversible perturbations that do not take the geometry into account. Moreover we demonstrate that irreversible perturbations generally can be implemented in conjunction with the stochastic gradient version of the Langevin algorithm. Lastly, while continuous-time irreversible perturbations cannot impair the performance of a Langevin estimator, the situation can sometimes be more complicated when discretization is considered. To this end, we describe a discrete-time example in which irreversibility increases both the bias and variance of the resulting estimator.

Most data is automatically collected and only ever "seen" by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, we characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations. Based on our theory, we design unsupervised objectives for training neural compressors. Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than $1000\times$ on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.

This paper describes the hierarchical infinite relational model (HIRM), a new probabilistic generative model for noisy, sparse, and heterogeneous relational data. Given a set of relations defined over a collection of domains, the model first infers multiple non-overlapping clusters of relations using a top-level Chinese restaurant process. Within each cluster of relations, a Dirichlet process mixture is then used to partition the domain entities and model the probability distribution of relation values. The HIRM generalizes the standard infinite relational model and can be used for a variety of data analysis tasks including dependence detection, clustering, and density estimation. We present new algorithms for fully Bayesian posterior inference via Gibbs sampling. We illustrate the efficacy of the method on a density estimation benchmark of twenty object-attribute datasets with up to 18 million cells and use it to discover relational structure in real-world datasets from politics and genomics.

Probabilistic Models in Machine Learning is the use of the codes of statistics to data examination. It was one of the initial methods of machine learning. It's quite extensively used to this day. Individual of the best-known algorithms in this group is the Naive Bayes algorithm. Probabilistic modelling delivers a framework for accepting what learning is.

For the binary prevalence quantification problem under prior probability shift, we determine the asymptotic variance of the maximum likelihood estimator. We find that it is a function of the Brier score for the regression of the class label against the features under the test data set distribution. This observation suggests that optimising the accuracy of a base classifier on the training data set helps to reduce the variance of the related quantifier on the test data set. Therefore, we also point out training criteria for the base classifier that imply optimisation of both of the Brier scores on the training and the test data sets.

This paper presents Bayesian parameter estimation for first order Grey system models' parameters (or sometimes referred to as hyperparameters). There are different forms of first-order Grey System Models. These include $GM(1,1)$, $GM(1,1| \cos(\omega t)$, $GM(1,1| \sin(\omega t)$, and $GM(1,1| \cos(\omega t), \sin(\omega t)$. The whitenization equation of these models is a first-order linear differential equation of the form \[ \frac{dx}{dt} + a x = f(t) \] where $a$ is a parameter and $f(t) = b$ in $GM(1,1|)$ , $f(t) = b_1\cos(\omega t) + b_2$ in $GM(1,1| cos(\omega t)$, $f(t) = b_1\sin(\omega t)+b_2$ in $GM(1,1| \sin(\omega t)$, $f(t) = b_1\sin(\omega t) + b_2\cos(\omega t) + b_3$ in $GM(1,1| \cos(\omega t), \sin(\omega t)$, $f(t) = b x^2$ in Grey Verhulst model (GVM), and where $b, b_1, b_2$, and $b_3$ are parameters. The results from Bayesian estimations are compared to the least square estimated models with fixed $\omega$. We found that using rolling Bayesian estimations for GM parameters can allow us to estimate the parameters in all possible forms. Based on the data used for the comparison, the numerical results showed that models with Bayesian parameter estimations are up to 45\% more accurate in mean squared errors.

We propose a very fast approximate Markov Chain Monte Carlo (MCMC) sampling framework that is applicable to a large class of sparse Bayesian inference problems, where the computational cost per iteration in several models is of order $O(ns)$, where $n$ is the sample size, and $s$ the underlying sparsity of the model. This cost can be further reduced by data sub-sampling when stochastic gradient Langevin dynamics are employed. The algorithm is an extension of the asynchronous Gibbs sampler of Johnson et al. (2013), but can be viewed from a statistical perspective as a form of Bayesian iterated sure independent screening (Fan et al. (2009)). We show that in high-dimensional linear regression problems, the Markov chain generated by the proposed algorithm admits an invariant distribution that recovers correctly the main signal with high probability under some statistical assumptions. Furthermore we show that its mixing time is at most linear in the number of regressors. We illustrate the algorithm with several models.

We present a new subspace-based method to construct probabilistic models for high-dimensional data and highlight its use in anomaly detection. The approach is based on a statistical estimation of probability density using densities of random subspaces combined with geometric averaging. In selecting random subspaces, equal representation of each attribute is used to ensure correct statistical limits. Gaussian mixture models (GMMs) are used to create the probability densities for each subspace with techniques included to mitigate singularities allowing for the ability to handle both numerical and categorial attributes. The number of components for each GMM is determined automatically through Bayesian information criterion to prevent overfitting. The proposed algorithm attains competitive AUC scores compared with prominent algorithms against benchmark anomaly detection datasets with the added benefits of being simple, scalable, and interpretable.

In machine learning and statistics, computing the accuracy, or loss, of a model is crucial for understanding the quality of the model and what improvements can be made to increase accuracy. Typically, researchers choose a loss function de- pending on their task, and this loss function runs over their test set of data, after training. However, in many cases, researchers want an estimation of their loss either before they test it or in cases when testing data is not yet available. This estimation is known as expected loss, or risk, and is usually utilized in order to assess how precarious an action or event will be. The foundations of Bayesian statistics are rooted in Bayes' Theorem, a theorem developed by Thomas Bayes who was an English mathematician and theologian during the 1700s.

Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data.