A Guide to Metaheuristic Optimization for Machine Learning Models in Python

Mathematical optimization is the process of finding the best set of inputs that maximizes (or minimizes) the output of a function. In the field of optimization, the function being optimized is called the objective function. A wide range of out-of-the-box tools exist for solving optimization problems that only work with well-behaved functions, also called convex functions. Well-behaved functions contain a single optimum, whether it is a maximum or a minimum value. Here a function can be thought of as a surface with a single valley (minimum) and/or hill (maximum).