The Impossibility of Parallelizing Boosting

Boosting is one of the most successful ideas in machine learning, allowing one to "boost" the performance of a base learning algorithm with rather poor accuracy into a highly accurate classifier, with recent applications in adversarial training [1], reinforcement learning [5], and federated learning [27], among many others. The classic boosting algorithm, known as AdaBoost [8], achieves this by iteratively training classifers on the training data set. After each iteration, the data set is reweighed and a new classifier is trained using a weighted loss function.