Throughout its history, Machine Learning (ML) has coexisted with Statistics uneasily, like an ex-boyfriend accidentally seated with the groom's family at a wedding reception: both uncertain where to lead the conversation, but painfully aware of the potential for awkwardness. This is caused in part by the fact that Machine Learning has adopted many of Statistics' methods, but was never intended to replace statistics, or even to have a statistical basis originally. Nevertheless, Statisticians and ML practitioners have often ended up working together, or working on similar tasks, and wondering what each was about. The question, "What's the difference between Machine Learning and Statistics?" has been asked now for decades. Machine Learning is largely a hybrid field, taking its inspiration and techniques from all manner of sources. It has changed directions throughout its history and often seemed like an enigma to those outside of it.1

We can see that there is a bleeding of ideas between fields and subfields in statistics. The machine learning practitioner must be aware of both the machine learning and statistical-based approach to the problem. This is especially important given the use of different terminology in both domains. In his course on statistics, Rob Tibshirani, a statistician who also has a foot in machine learning, provides a glossary that maps terms in statistics to terms in machine learning, reproduced below.

Since I approached Machine Learning during my Ph.D. in Statistics I've always tried to compare classical statistical approaches and machine learning ones. I mean, they are surely both fundamentally based on data and they both try to extract some kind of knowledge from data so where exactly is the difference? What is inherently different in those two fields? To answer these questions let's start from the very beginning: the definitions. Statistics is a traditional field, broadly defined as a branch of mathematics dealing with data collection, organization, analysis, interpretation and presentation (ref).