Regularization in deep learning – Chatbots Life

Part of the magic sauce for making the deep learning models work in production is regularization. For this blog post I'll use definition from Ian Goodfellow's book: regularization is "any modification we make to the learning algorithm that is intended to reduce the generalization error, but not its training error". For better theoretical understanding, I'd recommend checking out the chapter of the deep learning book dedicated to regularization. Generalization in machine learning refers to how well the concepts learned by the model apply to examples which were not seen during training. The goal of most machine learning models is to generalize well from the training data, in order to make good predictions in the future for unseen data. Overfitting happens when the models learns too well the details and the noise from training data, but it doesn't generalize well, so the performance is poor for testing data.