Weighted Random Search for Hyperparameter Optimization

The vast majority of machine learning algorithms involve two different sets of parameters: the training parameters and the meta-parameters (also known as hyperparameters). While the training parameters are learned during the training phase, the values of the hyperparameters have to be specified before the learning phase. For instance, the hyperparameters of neural networks typically specify the architecture of the network (number and type of layers, number and type of nodes, etc). Determining the optimal combination of hyperparameter values leading to the best generalization performance can be done through repeated training and evaluation sessions, trying different combinations of hyperparameter values. We call each training evaluation process for one combination of hyperparameter values a trial. Each trial is computationally expensive, since it involves retraining the model.