To Fuse or Not To Fuse Models?

The idea of model fusions is pretty simple: You combine the predictions of a bunch of separate classifiers into a single, uber-classifier prediction, in theory, better than the predictions of its individual constituents. As my colleague Teresa Álverez mentioned in a previous post, however, this doesn't typically lead to big gains in performance. In many cases, OptiML will find something as good or better than any combination you could try by hand. Why waste your time fiddling with combinations of models when you could spend it on doing things that will almost certainly have a more measurable impact on your model's performance, like feature engineering or better yet, acquiring more and better data? Part of the answer here is that looking at a number like "R squared" or "F1-score" is often an overly reductive view of performance.