Data preparation is the most time-consuming and hectic process in data science and machine learning, accounting for 80% of the labor. Messy data is a serious issue that costs businesses trillions of dollars every year. Model performance can be harmed by data errors (for example, mislabeled samples in the training set) and dataset-level concerns like overlapping classes. Most test set errors are ubiquitous even in gold-standard benchmark datasets. This can cause data scientists to deploy worse models. Although physically analyzing and cleaning up individual data points sounds tiresome, it frequently gives a significantly bigger payback than experimenting with advanced modeling approaches.

Distributed ML is an active area of work, in both academia and industry, and it has been for some time now. Companies like Google were doing distributed machine learning decades ago. For some use cases, libraries like scikit-learn are totally adequate, while for other use cases, e.g. when using sophisticated models that require a lot of compute to train, training over large datasets that don't fit on a single node, distributed computing is essential. On the topic of data storage: in some cases, system builders do co-design the data storage and data processing, e.g. Such co-design can give performance gains.