Deep Learning Book: Chapter 8-- Optimization For Training Deep Models Part I

When we don't have p_data but a finite training set, we have a ML problem. The latter can be converted back to an optimization problem by replacing p_data with the empirical distribution with p _data obtained from the training set, thereby reducing the empirical risk. Although this might look relatively similar to optimization, there are two main problems. Firstly, ERM is prone to overfitting with the possibility of the dataset being learned by high capacity models (models with the ability to learn extremely complex functions). Secondly, ERM might not be feasible.