Challenges and Opportunities in NLP Benchmarking

We thus need to rethink how we design our benchmarks and evaluate our models so that they can still serve as useful indicators of progress going forward. This post aims to give an overview of challenges and opportunities in benchmarking in NLP, together with some general recommendations. I tried to cover perspectives from recent papers, talks at ACL 2021 as well as at the ACL 2021 Workshop on Benchmarking: Past, Present and Future, in addition to some of my own thoughts.