Hyperparameter optimization in Python. Part 0: Introduction.

Hyperparameter optimization, or HPO as cool kids like to call it, is quickly becoming common knowledge in data science. Anything, with hyper in the name sounds cool enough, but what does it actually do and why should you care? Every machine learning algorithm has a certain number of parameters that you define before you start training. The number of layers, depth of a tree or the amount of regularization are just some examples of such (hyper)parameters. Once those are defined you can feed the data to your model, train it and evaluate its performance. Hyperparameter optimization is just the process of tweaking hyperparameters to achieve the highest performance under some time constraint.