Inverse Propensity Score based offline estimator for deterministic ranking lists using position bias

The mission of Just Eat Takeaway is to empower users' every food moment. A big part of fulfilling that mission is ensuring that we show the right restaurants to the right users. To do this, we design large-scale machine learning recommendation systems that can learn which types of users tend to enjoy which types of restaurants. A crucial part of this process is being able to compare the quality of different recommendation systems, in order to decide which is most effective. The gold standard approach to this evaluation problem is A/B testing - allow the different systems to recommend items to randomly selected groups of users, and compare their relative performance. However, A/B testing is both slow, taking time to reach sufficient statistical power, and expensive [5, 1] if we serve poor recommendations to a subset of users. Therefore, evaluating recommender systems in offline fashion without running A/B tests is of significant importance. These offline approaches avoid serving experimental recommenders to users, and instead evaluates them using old online data, where users were served recommendations from a different system. Metrics such as Root Mean Squared Error [2] Mean Average Precision, Normalized Discounted Cumulative Gain, Mean Reciprocal Rank and Hit Rate have been used repeatedly for offline evaluation.