Sharded: A New Technique To Double The Size Of PyTorch Models

For those with not much time to read through the intuition of how sharded works, I'm going to explain up-front how to use sharded with your PyTorch code. But I encourage you to read through the end of the article to understand how sharded works. Sharded is meant to be used with multiple GPUs to gain all the benefits. But, training on multiple GPUs can be intimidating and a huge pain to set up. The easiest way to supercharge your code with sharded is to convert your model to PyTorch Lightning (which is just a simple refactor).