We make Bayesian Additive Regression Networks (BARN) available as a Python package, \texttt{barmpy}, with documentation at \url{https://dvbuntu.github.io/barmpy/} for general machine learning practitioners. Our object-oriented design is compatible with SciKit-Learn, allowing usage of their tools like cross-validation. To ease learning to use \texttt{barmpy}, we produce a companion tutorial that expands on reference information in the documentation. Any interested user can \texttt{pip install barmpy} from the official PyPi repository. \texttt{barmpy} also serves as a baseline Python library for generic Bayesian Additive Regression Models.

Scikit-Learn offers two vehicles for optimizing hyperparameter tuning: GridSearchCV and RandomizedSearchCV. GridSearchCV performs an exhaustive search over specified parameter values for an estimator (or machine learning algorithm) and returns the best performing hyperparametric combination. So, all we need to do is specify the hyperparameters with which we want to experiment and their range of values, and GridSearchCV performs all possible combinations of hyperparameter values using cross-validation. As such, we naturally limit our choice of hyperparameters and their range of values. Theoretically, we can specify a set of parameter values for ALL hyperparameters of a model, but such a search consumes vast computer resources and time.

Hyperparameters are parameters that are defined before training to specify how we want model training to happen. We have full control over hyperparameter settings and by doing that we control the learning process. For example in the random forest model n_estimators (number of decision trees we want to have) is a hyperparameter. It can be set to any integer value but of course, setting it to 10 or 1000 changes the learning process significantly. Parameters, on the other hand, are found during the training. We have no control over parameter values as they are the result of model training.

In my last article, I introduced you to the world of marketing mix modeling. If you have not read it so far, please do before you proceed. There, we have a created a linear regression model that is able to predict sales based on raw advertising spends in several advertising channels, such as TV, radio, web banners. For me as a machine learning practitioner, such a model is nice already on its own. Even better, it also makes business people happy because the model lets us calculate ROIs, allowing us to judge how well each channel performed.

Grid Search can be thought of as an exhaustive search for selecting a model. In Grid Search, the data scientist sets up a grid of hyperparameter values and for each combination, trains a model and scores on the testing data. In this approach, every combination of hyperparameter values is tried which can be very inefficient. For example, searching 20 different parameter values for each of 4 parameters will require 160,000 trials of cross-validation. This equates to 1,600,000 model fits and 1,600,000 predictions if 10-fold cross validation is used.

For the python code, I used the Iris dataset which is available within the Scikit-learn package. It is a very small dataset (150 rows only) with a multiclass classification problem. As we are mostly focussing on hyperparameter tuning, I have not performed the EDA(exploratory data analysis) or feature engineering part and directly jumped into the model-building. I used the XGBoostClssifier algorithm for the model-building to classify the target variables. GridSearchCV is a function that comes in Scikit-learn's(or SKlearn) model_selection package.To use the GridSearchCV function, first, we define a dictionary in which we mention a particular hyperparameter along with the values it can take.

Optimizing hyperparameters for machine learning models is a key step in making accurate predictions. Hyperparameters define characteristics of the model that can impact model accuracy and computational efficiency. They are typically set prior to fitting the model to the data. In contrast, parameters are values estimated during the training process that allow the model to fit the data. Hyperparameters are often optimized through trial and error; multiple models are fit with a variety of hyperparameter values, and their performance is compared. For random forest algorithms, one can manipulate a variety of key attributes that define model structure.

I have included Python code in this article where it is most instructive. Full code and data to follow along can be found on the project Github page. The best way to think about hyperparameters is like the settings of an algorithm that can be adjusted to optimize performance, just as we might turn the knobs of an AM radio to get a clear signal (or your parents might have!). While model parameters are learned during training -- such as the slope and intercept in a linear regression -- hyperparameters must be set by the data scientist before training. In the case of a random forest, hyperparameters include the number of decision trees in the forest and the number of features considered by each tree when splitting a node.

Previously, we wrote about some common trade-offs in machine learning and the importance of tuning models to your specific dataset. We demonstrated how to tune a random forest classifier using grid search, and how cross-validation can help avoid overfitting when tuning hyperparameters (HPs). You'll learn a different strategy for traversing hyperparameter space - randomized search - and how to use it to tune two other classification algorithms - a support vector machine and a regularized logistic regression classifier. We'll keep working with the wine dataset, which contains chemical characteristics of wines of varying quality. As before, our goal is to try to predict a wine's quality from these features.