I do still have some concerns regarding applicability, since a gold standard "ground truth" of inference is required. I do appreciate the situation described in the feedback, where one is trying to decide on some approximate algorithm to "deploy" in the wild, is actually fairly common. That is, the sort of setting where very slow MCMC can be run for a long time on training data, but on new data, where there is e.g. a real-time requirement, a faster approximate inference algorithm will be used instead. The approach is based on constructing an estimator of the "symmetric" KL divergence (i.e., the sum of the forward and reverse KL) between an approximation to the target distribution and a representation of the "true" exact target distribution. The overall approach considered is interesting, and for the most part clearly presented.

Not all data are created equal. But how much information is any piece of data likely to contain? This question is central to medical testing, designing scientific experiments, and even to everyday human learning and thinking. MIT researchers have developed a new way to solve this problem, opening up new applications in medicine, scientific discovery, cognitive science, and artificial intelligence. In theory, the 1948 paper, "A Mathematical Theory of Communication," by the late MIT Professor Emeritus Claude Shannon answered this question definitively.

Approximate probabilistic inference algorithms are central to many fields. Examples include sequential Monte Carlo inference in robotics, variational inference in machine learning, and Markov chain Monte Carlo inference in statistics. A key problem faced by practitioners is measuring the accuracy of an approximate inference algorithm on a specific data set. This paper introduces the auxiliary inference divergence estimator (AIDE), an algorithm for measuring the accuracy of approximate inference algorithms. AIDE is based on the observation that inference algorithms can be treated as probabilistic models and the random variables used within the inference algorithm can be viewed as auxiliary variables.

Lifted inference algorithms for representations that combine first-order logic and probabilistic graphical models have been the focus of much recent research. All lifted algorithms developed to date are based on the same underlying idea: take a standard probabilistic inference algorithm (e.g., variable elimination, belief propagation etc.) and improve its efficiency by exploiting repeated structure in the first-order model. In this paper, we propose an approach from the other side in that we use techniques from logic for probabilistic inference. In particular, we define a set of rules that look only at the logical representation to identify models for which exact efficient inference is possible. We show that our rules yield several new tractable classes that cannot be solved efficiently by any of the existing techniques.