Metrology for AI: From Benchmarks to Instruments

Chris Welty, Praveen Paritosh, Lora Aroyo Google Research Abstract In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations. We begin with the intuitive observation that evaluating the performance of an AI system is a form of measurement. In all other science and engineering disciplines, the devices used to measure are called instruments, and all measurements are recorded with respect to the characteristics of the instruments used. One does not report mass, speed, or length, for example, of a studied object without disclosing the precision (measurement variance) and resolution (smallest detectable change) of the instrument used. It is extremely common in the AI literature to compare the performance of two systems by using a crowd-sourced dataset as an instrument, but failing to report if the performance difference lies within the capability of that instrument to measure. To illustrate the adoption of metrology to benchmark datasets we use the word similarity benchmark WS353 and several previously published experiments that use it for evaluation. 1 Contributions of this paper In this paper we examine the question of how the variations in human interpretation and other aspects of data collection can affect the measurements we make with crowd-powered datasets. For this, we adopt metrology, the science of measurement and its application, and apply it to human (crowd) powered evaluations. We begin with the intuitive observation that evaluating the performance of an AI system is a form of measurement. In all other science and engineering disciplines, the devices used to measure are called instruments, and all measurements are recorded with respect to the characteristics of the instruments used. One does not report mass, speed, or length, for example, of a studied object without disclosing the precision (measurement variance) and sensitivity (smallest detectable change) of the instrument used. This dataset has been cited over 1500 times, and has spurred the development and evaluation of automated approaches to computing lexical/semantic similarity (Witten and Milne 2008; Agirre et al. 2009) and word embeddings (Mitchell and Lapata 2008; Mikolov et al. 2013; Levy, Goldberg, and Dagan 2015; Pennington, Socher, and Manning 2014; Bojanowski et al. 2017).