Debiased Offline Evaluation of Active Learning in Recommender Systems

Active Learning (AL) when applied to Recommender Systems (RSs) aims at proactively acquiring additional ratings data from the RS users in order to improve subsequent recommendation quality. AL strategies are typically evaluated offline first, but the classic AL offline evaluation methodology does not take into account the bias problem in RS offline evaluation. This problem affects the evaluation of an RS, as brought to light by recent literature. But, we argue, it also affects the evaluation of AL strategies as well. For this reason, in this paper, we propose a new AL offline evaluation methodology for RSs which mitigates the bias and thus facilitates a truer picture of the performances of the AL strategies under evaluation. We illustrate our proposed methodology on two datasets and with three simple and well-known AL strategies from the literature. Our experimental results differ from those reported previously in the literature, which shows the importance of our approach to AL evaluation.