Reviews: Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach

This paper builds on the work of Platanios et al. (2014, 2016) on estimating the accuracy of a set of classifiers for a given task using only unlabeled data, based on the agreement behavior of the classifiers. The current work uses a probabilistic soft logic (PSL) model to infer the error rates of the classifiers. The paper also extends this approach to the case where we have multiple related classification tasks: for instance, classifying noun phrases with regard to their membership in multiple categories, some of which subsume others and some of which are mutually exclusive. The paper shows how a PSL model can take into account these constraints among the categories, yielding better error rate estimates and higher joint classification accuracy. It is well written and the methodology seems sound.