Generative modeling through internal high-dimensional chaotic activity

Generative models aim to create samples statistically similar to those belonging to a training dataset: their goal is to fit the probability distribution from which the datapoints supposedly come from. In a generic setting, this probability distribution takes the form of a Boltzmann factor. The corresponding Energy Based Models (EBMs) fit the parameters of the Hamiltonian of the Boltzmann distribution and can be viewed as maximum entropy models, where the statistical properties of the dataset are imposed as constraints to low degree correlation functions [1, 2], (see [3, 4] for recent reviews). The resulting learning rule can be viewed as a gradient ascent on the Log-Likelihood (LL). However, running the training dynamics is a notoriously challenging task: at each epoch, the evaluation of the gradient of the LL requires the computation of the correlation functions of the degrees of freedom as predicted from the current estimation of the model's probability distribution. This is typically an intractable problem from an analytical point of view and is generally tackled numerically through parallel Monte Carlo Markov Chain (MCMC) simulations.