Online evaluation of machine learning models - O'Reilly Artificial Intelligence Conference in London 2019

Academic machine learning involves almost exclusively offline evaluation of machine learning models. In the real world this is, somewhat surprisingly, often only good enough for a rough cut that eliminates the real dogs. For production work, online evaluation is often the only option to determine which of several final-round candidates might be chosen for further use. As Einstein is rumored to have said, theory and practice are the same, in theory. So it is with models.