GitHub - alpa-projects/alpa: Training and serving large-scale neural networks

Alpa is a system for training and serving large-scale neural networks. Scaling neural networks to hundreds of billions of parameters has enabled dramatic breakthroughs such as GPT-3, but training and serving these large-scale neural networks require complicated distributed system techniques. Alpa aims to automate large-scale distributed training and serving with just a few lines of code. Alpa automatically parallelizes users' single-device code on distributed clusters with data, operator, and pipeline parallelism. Alpa achieves linear scaling on training models with billions of parameters on distributed clusters.