Using compute power to iterate faster through ML experiments

Waiting for scripts to terminate is an absolute pet peeve of mine. Not only is it a waste of time but the constant context-switching is distracting and tiring. One grievous offender is the ML experiment, which lasts anywhere from a few minutes to a few days. A fixture of the development lifecycle, the ML experiment takes place any time you want to evaluate your model on a new set of hyperparameters (as in hyperparameter optimization), explore new features (feature selection), or try out a new model architecture. In a way, features and model architectures are simply glorified forms of hyperparameters.