Motif of the Mind

This blog post is inspired by a user question on Discourse. The ability to predict new data from old observations has long been considered as one of the golden rules of evaluating science and scientific theory. And in Bayesian modelling, this idea is especially natural: not only it maps new inputs into new outputs the same way as a deterministic model, it does so probabilistically, meaning that you also get the uncertainty of each prediction. Consider a linear regression problem: the data could be represented as a tuple ($X$, $y$) and we want to find the linear relationship which maps $X\to y$. A subtle point here to note here is that values in $X$ are usually considered as given, something trivial to measure, or has little noise (even noiseless).