Multi-GPU Training in Pytorch

Let's say you have 3 GPUs available and you want to train a model on one of them. To allow Pytorch to "see" all available GPUs, use: There are a few different ways to use multiple GPUs, including data parallelism and model parallelism. Data parallelism refers to using multiple GPUs to increase the number of examples processed simultaneously. For example, if a batch size of 256 fits on one GPU, you can use data parallelism to increase the batch size to 512 by using two GPUs, and Pytorch will automatically assign 256 examples to one GPU and 256 examples to the other GPU. Using data parallelism can be accomplished easily through DataParallel.