BitTorrent For Machine Learning: Now Use Supercomputers From Home

Training the popular GPT-3 from scratch can cost millions of dollars. But, what if an individual researcher wants to experiment on such a large scale? It is almost impossible to generate funds for toying with networks. To address this large gap between innovation and computation in ML, a team of researchers from Russia have introduced Learning@home -- a neural network training paradigm that handles large amounts of poorly connected participants. "Hypothetically, a researcher could crowdsource the training of large neural networks with thousands of regular PCs provided by volunteers. The raw computing power of a hundred thousand $2500 desktops dwarfs that of a $250M server pod," wrote the researchers.