How to evaluate machine learning? U of T research supports latest benchmark initiative

Machine learning, a popular subfield of artificial intelligence that is revolutionizing everything from legal research to medical diagnostics, depends on three major parts: a model, a dataset, and the hardware that it's backed by. So how do researchers, startups and companies evaluate its overall effectiveness? Options were limited until the recent formation of MLPerf, a consortium of industry and academic partners including Google, Intel, Baidu, Harvard University, Stanford University and the University of Toronto, who are working together to offer a new benchmark suite to evaluate machine learning (ML) performance, from speed to system cost and power efficiency. "Current benchmark suites give some basic numbers to say how well these benchmarks perform on certain hardware, but do not provide any insight into why these applications perform one way or another," says Gennady Pekhimenko, an assistant professor in the department of computer and mathematical sciences at U of T Scarborough and the tri-campus graduate department of computer science. "To know which design decision is bad or not for ML applications, you want to have some representative reference model," he says.