Adaptive Parameters Methods for Machine Learning

In this post, I will discuss the ideas behind adaptive parameters methods for machine learning and why and when to implement them as some practical examples using python. Adaptive methods (also known as parameter scheduling) refer to strategies to update some model parameters at training time using a schedule. This change will depend on the model's state at time t; for example, you can update them depending on the loss value, the number of iterations/epochs, elapsed training time, etc. For example, in general, for neural networks, the choice of the learning rate has several consequences; if the learning rate is too large, it may overshoot the minimum; if it's too small, it may take it too long to converge, or it might get stuck on a local minimum. In this scenario, we choose to change the learning rate as a function of the epochs; this way, you may set a large rate at the beginning of the training, and conforming to the epochs increases; you can decrease the value until you reach a lower threshold.