Machine learning is getting BIG (Part II)

This article is a continuation of'Machine learning is getting BIG (Part I)' previously published on this blog. As well as cost in terms of dollars, people have started calculating the cost of training these huge models in terms of greenhouse gas emissions. When neural architecture search was included, training a single'transformer' model was found to generate more carbon dioxide than 4 lifetimes of car use. It is true that modern hardware is much more efficient than older hardware. Although the improvement in efficiency has not tracked the increase in use of energy on these projects. Those are the run time costs.