PyTorch – Internal Architecture Tour

This post is a tour around the PyTorch codebase, it is meant to be a guide for the architectural design of PyTorch and its internals. My main goal is to provide something useful for those who are interested in understanding what happens beyond the user-facing API and show something new beyond what was already covered in other tutorials. Note: PyTorch build system uses code generation extensively so I won't repeat here what was already described by others. If you're interested in understanding how this works, please read the following tutorials: As you probably know, you can extend Python using C and C and develop what is called as "extension". All the PyTorch heavy work is implemented in C/C instead of pure-Python.