Why We Need New Benchmarks for AI

How do you measure artificial intelligence? Since the idea first took hold in the 1950s, researchers have gauged the progress of AI by establishing benchmarks, such as the ability to recognize images, create sentences and play games like chess. These benchmarks have proved a useful way to determine whether AI is better able to do more things--and to drive researchers toward creating AI tools that are even more useful. In the past few years, AI systems have surpassed many of the tests researchers have proposed, beating humans at many tasks. For researchers, the mission now is to create benchmarks that could capture the broader kinds of intelligence that could make AI truly useful--benchmarks, for instance, that can reflect elusive skills such as reasoning, creativity and the ability to learn.