Smarter Parameter Sweeps (or Why Grid Search Is Plain Stupid)

Anyone that ever had to train a machine learning model had to go through some parameter sweeping (a.k.a. For random forests the parameters in need of optimization could be the number of trees in the model and the number of features considered at each split, for a neural network, there is the learning rate, the number of hidden layers, the number of hidden units in each layer, and several other parameters. Hyper-parameter optimization requires the use (and maybe the abuse) of a validation set on which you can't trust your performance metrics anymore. In this sense it is like a second phase of learning, or an extension to the learning algorithm itself. The performance metric (or the objective function) can be visualized as a heat-map in the n-dimensional parameter-space or as a surface in an n 1-dimensional space (the dimension n 1 being the value of that objective function).