Impact of HPO on AutoML Forecasting Ensembles

Due to this uncertainty over which models will perform best, it is common place in the forecasting space that domain experts and data scientists have to experiment with several methods before they find one that works acceptably well on a particular problem. This exploration process can be time and resource consuming and is not always practical, due to the plethora of unsolved forecasting problems as well as the scarcity of domain experts and data scientists. In recent years, Automated Machine Learning (AutoML) has become more popular, allowing non-technical users to solve machine learning problems without in depth knowledge about the underlying methodology, filling in for the lack of available data scientists through automation [5]. In forecasting there are several approaches to AutoML, one of them being the established method of using ensemble learning and aggregation of forecasts [6]. This has seen a recent increase in attention, with the top performing models in the M4 Competition [7] being of this nature [8]. Ensembling, can be conceptualised as the automation of the previously manual step of exploring the performance of various algorithms on a given problem and selecting the best one or a combination of models. This, however, does not address another important aspect of data science which is the selection of good hyperparameters, leading to better performance of a model trained using a particular algorithm. The combination of ensemble learning and hyperparameter tuning in an AutoML forecasting setup will be discussed in this paper.