Guaranteeing Reproducibility in Deep Learning Competitions

Democratizing access to artificial intelligence (AI) requires competitions that promote the development of sample-efficient learning, as well as ensure the reproducibility and generalizability of results. Sample efficiency is important because practitioners with limited compute resources cannot readily utilize algorithms that require a massive number of samples. The complexity of these stateof-the-art methods is outpacing advancements in computation. Moreover, as methods and domains become more specialized, learning procedures become more fragile: often undocumented modifications can inhibit reproducible results and seeds are chosen to reflect the optimal performance of a given solution [Henderson et al., 2018]. Because the focus of traditional research challenges is the development of new techniques in a particular field, these challenges seek to reward participants for novel solutions. However, submissions with the best performance on the (often highly specified) task tend leverage domain knowledge that is not broadly applicable, leading challenges to open separate tracks where submissions are subjectively evaluated on research novelty [Pavlov et al., 2018]. To encourage participants to develop methods with reproducible and robust training behavior, we propose a challenge paradigm where competitors are evaluated directly on the performance of their learning procedures rather than pre-trained agents. Since competition organizers retrain submissions in a controlled setting they can guarantee reproducibility, and - by retraining submissions using a held-out test set - help ensure generalization of submissions past the environments on which they were trained.