BaguaSys/Bagua is a deep learning acceleration framework for PyTorch developed by AI platform@Kuaishou Technology and DS3 Lab@ETH. Bagua supports multiple, advanced distributed training algorithms with state-of-the-art system relaxation techniques. These include quantization, decentralization, and communication delay. Bagua generally produces a higher training throughput than vanilla PyTorch DistributedDataParallel (the de-facto strategy) due to their custom techniques and is written in Rust. You can check out this paper for more details on the Bagua system, as well as this book to learn more about the theoretical guarantees of the involved algorithms. Bagua thrives on the diversity of distributed communication algorithms.

Learn enough Lightning to match the level of expertise required by your research or job. Researchers and machine learning engineers should start here. Add validation and test sets to avoid over/underfitting. Add parameters to your script so you can run from the commandline. Learn to scale up your models and enable collaborative model development at academic or industry research labs.