Over-fitting and Regularization – Towards Data Science – Medium

In supervised machine learning, models are trained on a subset of data aka training data. The goal is to compute the target of each training example from the training data. Now, overfitting happens when model learns signal as well as noise in the training data and wouldn't perform well on new data on which model wasn't trained on. In the example below, you can see underfitting in first few steps and overfitting in last few. Now, there are few ways you can avoid overfitting your model on training data like cross-validation sampling, reducing number of features, pruning, regularization etc. Regularization basically adds the penalty as model complexity increases.