Bagua: A new, efficient, distributed training strategy available in PyTorch Lightning 1.6

BaguaSys/Bagua is a deep learning acceleration framework for PyTorch developed by AI platform@Kuaishou Technology and DS3 Lab@ETH. Bagua supports multiple, advanced distributed training algorithms with state-of-the-art system relaxation techniques. These include quantization, decentralization, and communication delay. Bagua generally produces a higher training throughput than vanilla PyTorch DistributedDataParallel (the de-facto strategy) due to their custom techniques and is written in Rust. You can check out this paper for more details on the Bagua system, as well as this book to learn more about the theoretical guarantees of the involved algorithms. Bagua thrives on the diversity of distributed communication algorithms.