Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements

Evaluation is a key part of machine learning (ML), yet there is a lack of support and tooling to enable its informed and systematic practice. We introduce Evaluate and Evaluation on the Hub--a set of tools to facilitate the evaluation of models and datasets in ML. Evaluate is a library to support best practices for measurements, metrics, and comparisons of data and models. Its goal is to support reproducibility of evaluation, centralize and document the evaluation process, and broaden Figure 1: Average number of evaluation datasets and evaluation to cover more facets of model metrics per paper, based on 10 random samples per performance. It includes over 50 efficient year from EMNLP proceedings over the past two canonical implementations for a variety of domains decades. More recent papers use more datasets and and scenarios, interactive documentation, metrics, while fewer of them report statistical significance and the ability to easily share implementations test results.