Despite being confined within the interior darkness of the skull, the human brain possesses a remarkable ability to interpret, understand and analyse the world out there, plan for unseen futures, and make decisions that can alter the course of events. This extraordinary capability is conjectured to come from the brain's function as a predictive machine, constantly inferring the hidden causes of its sensory inputs to maintain a coherent model of its environment. This view, which dates back to Helmholtz's idea of "perception as unconscious inference" (von Helmholtz, 1866)--evolving into the "Bayesian brain" hypothesis (Doya et al., 2007)--suggests that the brain operates as a constructive statistical organ. It updates its beliefs about the external world based on incoming sensory data under a generative model (GM). The GM furnishes the brain with a structured representation that supports probabilistic beliefs over both the latent dynamical states of the external world, corresponding to the generative process (GP), as well as the observation mappings through which these states give rise to sensory signals. Essentially, the brain continually refines its probabilistic beliefs about both the latent states and the causal mechanisms of the world through a process of online triple estimation, jointly optimising beliefs over: hidden states, model parameters, and their associated uncertainties in accordance with the principles of Bayesian inference (Eells, 2004; Parr et al., 2022). More technically, given a sensory observation yt at time t, perception can be formulated as an online triple estimation scheme, whose three components are: 1) online hidden state inference, 2) online parameter learning, and 3) online uncertainty estimation, all three of which are the core components of our proposed online generalised PC scheme and are elaborated in Section.

Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.

Thank you all for your helpful comments on our Comp Neuro paper. If the results of Figure 1 are indicative, this could further improve the results. The supervised training phase is depicted in the somewhat busy Fig. S2. While we disagree with Reviewer #2's opinion that the connection between neural regression and GPs is completely

Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.

Bayesian model reduction provides an efficient approach for comparing the performance of all nested sub-models of a model, without re-evaluating any of these sub-models. Until now, Bayesian model reduction has been applied mainly in the computational neuroscience community on simple models. In this paper, we formulate and apply Bayesian model reduction to perform principled pruning of Bayesian neural networks, based on variational free energy minimization. Direct application of Bayesian model reduction, however, gives rise to approximation errors. Therefore, a novel iterative pruning algorithm is presented to alleviate the problems arising with naive Bayesian model reduction, as supported experimentally on the publicly available UCI datasets for different inference algorithms. This novel parameter pruning scheme solves the shortcomings of current state-of-the-art pruning methods that are used by the signal processing community. The proposed approach has a clear stopping criterion and minimizes the same objective that is used during training. Next to these benefits, our experiments indicate better model performance in comparison to state-of-the-art pruning schemes.

Even though the brain operates in pure darkness, within the skull, it can infer the most likely causes of its sensory input. An approach to modelling this inference is to assume that the brain has a generative model of the world, which it can invert to infer the hidden causes behind its sensory stimuli, that is, perception. This assumption raises key questions: how to formulate the problem of designing brain-inspired generative models, how to invert them for the tasks of inference and learning, what is the appropriate loss function to be optimised, and, most importantly, what are the different choices of mean field approximation (MFA) and their implications for variational inference (VI).

Gaussian processes (GPs) are flexible distributions over functions that enable high-level assumptions about unknown functions to be encoded in a parsimonious, flexible and general way. Although elegant, the application of GPs is limited by computational and analytical intractabilities that arise when data are sufficiently numerous or when employing non-Gaussian models. Consequently, a wealth of GP approximation schemes have been developed over the last 15 years to address these key limitations. Many of these schemes employ a small set of pseudo data points to summarise the actual data. In this paper, we develop a new pseudo-point approximation framework using Power Expectation Propagation (Power EP) that unifies a large number of these pseudo-point approximations. Unlike much of the previous venerable work in this area, the new framework is built on standard methods for approximate inference (variational free-energy, EP and Power EP methods) rather than employing approximations to the probabilistic generative model itself. In this way, all of approximation is performed at `inference time' rather than at `modelling time' resolving awkward philosophical and empirical questions that trouble previous approaches. Crucially, we demonstrate that the new framework includes new pseudo-point approximation methods that outperform current approaches on regression and classification tasks.

Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.