1c446a652e50b1ea5618b66c07bfc0c5-Supplemental-Conference.pdf

While other areas of machine learning have seen more and more automation, designing a high-performing recommender system still requires a high level of human effort. Furthermore, recent work has shown that modern recommender system algorithms do not always improve over well-tuned baselines. A natural follow-up question is, "how do we choose the right algorithm for anew dataset and performance metric?" In this work, we start by giving the first large-scale study ofrecommender system approaches bycomparing 24algorithms and100 sets of hyperparameters across 85 datasets and 315 metrics. We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric.