How 3DFY.AI Built a Multi-Cloud, Distributed Training Platform Over Spot Instances with…

Deep Learning development is becoming more and more about minimizing the time from idea to trained model. To shorten this lead time, researchers need access to a training environment that supports running multiple experiments concurrently, each utilizing several GPUs. Until recently, training environments with tens or hundreds of GPUs were the sole property of the largest and richest technology companies. However, recent advances in the open-source community have helped close this gap, making this technology accessible even for small startups. In this series, we will share our experience in building out a scalable training environment using TorchElastic and Kubernetes, utilizing spot instances and deployed in multiple cloud providers.