Optimizing hyperparameters is one of the key elements of the life cycle of machine learning solutions. Yet, the processes for hyperparameter optimization remain incredibly laborious and require considerable effort from data scientists. Lately, there a new generation of tools and platforms have emerged with the focus of streamlining the experience of optimizing hyperparameters. Building deep learning solutions in the real world is a process of constant experimentation and optimization. Differently from any other type of software application, deep learning applications don't have a linear lifecycle based on the fact that models need to constantly refined, optimized and tested.

Machine learning models are often pre-set with specific parameters for easy implementation.

Machine learning models are often pre-set with specific parameters for easy implementation.

Preferred Networks has released a beta version of an open-source, automatic hyperparameter optimization framework called Optuna. In this blog, we will introduce the motivation behind the development of Optuna as well as its features. A hyperparameter is a parameter to control how a machine learning algorithm behaves. In deep learning, the learning rate, batch size, and number of training iterations are hyperparameters. Hyperparameters also include the numbers of neural network layers and channels.