Machine learning, and especially deep learning, has become increasingly more accurate in the past few years. This has improved our lives in ways we couldn't imagine just a few years ago, but we're far from the end of this AI revolution. Cars are driving themselves, x-ray photos are being analyzed automatically, and in this pandemic age, machine learning is being used to predict outbreaks of the disease, help with diagnosis, and make other critical healthcare decisions. And for those of us who are sheltering at home, recommendation engines in video on-demand platforms help us forget our troubles for an hour or two. This increase in accuracy is important to make AI applications good enough for production, but there has been an explosion in the size of these models.

In our previous post, we talked about how red AI means adding computational power to "buy" more accurate models in machine learning, and especially in deep learning. We also talked about the increased interest in green AI, in which we not only measure the quality of a model based on accuracy but also how big and complex it is. We covered different ways of measuring model efficiency and showed ways to visualize this and select models based on it. Maybe you also attended the webinar? If not, take a look at the recording where we also cover a few of the points we'll describe in this blog post.

Since 2012, the field of artificial intelligence (AI) has reported remarkable progress on a broad range of capabilities including object recognition, game playing, speech recognition, and machine translation.43 Much of this progress has been achieved by increasingly large and computationally intensive deep learning models.a Figure 1, reproduced from Amodei et al.,2 plots training cost increase over time for state-of-the-art deep learning models starting with AlexNet in 201224 to AlphaZero in 2017.45 The chart shows an overall increase of 300,000x, with training cost doubling every few months. An important paper47 has estimated the carbon footprint of several NLP models and argued this trend is both environmentally unfriendly and prohibitively expensive, raising barriers to participation in NLP research. We refer to such work as Red AI. The amount of compute used to train deep learning models has increased 300,000x in six years. Figure taken from Amodei et al.2 This trend is driven by the strong focus of the AI community on obtaining "state-of-the-art" results,b as exemplified by the popularity of leaderboards,53,54 which typically report accuracy (or other similar measures) but omit any mention of cost or efficiency (see, for example, leaderboards.allenai.org).c Despite the clear benefits of improving model accuracy, the focus on this single metric ignores the economic, environmental, and social cost of reaching the reported results.

Like many of you in Towards Data Science, I am self-taught in machine learning and can remember the day I first left my laptop in model training, fan whirring away, to return hours later and find it still buzzing along. "Hmm", I thought, "maybe I need a GPU". Fellow DIY thinkers: it is not a good idea to try and rip apart an old Xbox, but you could build your own. On days even further in the past, energy usage for my family sometimes created a choice between playing video games and air conditioning: rainy summer afternoons with the Xbox fan and the additional external Xbox fan both battling for sound preeminence with video games and other electronic weapons. Those were the days before discussions about energy efficiency triggered a climate change debates.