Optimal Belief Approximation

In Bayesian statistics, probabilities are interpreted as degrees of belief. For any set of mutually exclusive and exhaustive events, one expresses the state of knowledge as a probability distribution over that set. The probability of an event then describes the personal confidence that this event will happen or has happened. As a consequence, probabilities are subjective properties reflecting the amount of knowledge an observer has about the events; a different observer might know which event happened and assign different probabilities. If an observer gains information, she updates the probabilities she had assigned before. If the set of possible mutually exclusive and exhaustive events is infinite, it is generally impossible to store all entries of the corresponding probability distribution on a computer or communicate it through a channel with finite bandwidth. One therefore needs to approximate the probability distribution which describes one's belief. Given a limited set X of approximative beliefs q(s) on a quantity s, what is the best belief to approximate the actual belief as expressed by the probability p(s)? In the literature, it is sometimes claimed that the best approximation is given by the q X that minimizes the Kullback-Leibler divergence ("approximation" KL) [1] KL(p, q) () p(s) p(s) ln (1) q(s)