Ease of use, expressivity, and debuggability are among the core principles of PyTorch. One of the key drivers for the ease of use is that PyTorch execution is by default "eager, i.e. op by op execution preserves the imperative nature of the program. However, eager execution does not offer the compiler based optimization, for example, the optimizations when the computation can be expressed as a graph. LazyTensor [1], first introduced with PyTorch/XLA, helps combine these seemingly disparate approaches. While PyTorch eager execution is widely used, intuitive, and well understood, lazy execution is not as prevalent yet.

PyTorch is an open source machine learning framework, primarily developed by Meta (previously Facebook). PyTorch is extensively used in the research space and in recent years it has gained immense traction in the industry due to its ease of use and deployment. Vertex AI, a fully managed end-to-end data science and machine learning platform on Google Cloud, has first class support for PyTorch making it optimized, compatibility tested and ready to deploy. We started a new blog series - PyTorch on Google Cloud - to uncover, demonstrate and share how to build, train and deploy PyTorch models at scale on Cloud AI Infrastructure using GPUs and TPUs on Vertex AI, and how to create reproducible machine learning pipelines on Google Cloud . This blog post is the home page to the series with links to the existing and upcoming posts for the readers to refer to.

A version of the PyTorch machine learning framework that incorporates a deep learning compiler to connect the Python package to cloud Tensor processors (TPUs) is now available on Google Cloud, the public cloud vendor and PyTorch co-developer Facebook announced. The general availability on PyTorch/XLA means users can access cloud TPU accelerators via a stable integration, the companies said Tuesday (Sept. Separately, promoters of the programming language R released a package that allows developers to use "PyTorch functionality natively from R." The new tool, dubbed "Torch for R," requires no Python installation. Meanwhile, Facebook and Google said PyTorch/XLA combines the machine learning library's APIs with XLA's linear algebra compiler that targets CPUs, GPUS and, now, cloud TPUs. While running on most standard Python programs, PyTorch/XLA defaults to CPUs for operations not yet supported on Tensor processors. That framework helps PyTorch users "find bottlenecks and adapt their programs to run more efficiently on cloud TPUs," said Craig Wiley, director of product development for Google's Cloud AI platform.

At the time of writing these lines running PyTorch code on TPUs is not a well-trodden path. Naturally, TPUs have been optimized for and mainly used with TensorFlow. But Kaggle and Google distribute free TPU time on some of its competitions, and one doesn't simply change his favorite framework, so this is a memo on my (mostly successful) experience of training PyTorch models with TPU on GCP. PyTorch/XLA is the project that allows doing it. It is still in active development, issues get fixed.