Facebook is easing 3D deep learning woes, one solution at a time. Last year, it announced Mesh R-CNN, a system that could render 3D objects from 2D shapes, and this year it has unveiled PyTorch3D. Conventional methods could not give apt solutions. PyTorch3D fulfills the above two shortages. It is an optimized and highly modular library.

Facebook AI Research (FAIR) today unveiled PyTorch3D, a library that enables researchers and developers to combine deep learning and 3D objects. As part of the release, Facebook is also open-sourcing Mesh R-CNN, a model introduced last year capable of rendering 3D objects from 2D shapes in images of interior spaces. PyTorch3D was inspired by Mesh R-CNN and recent 3D work by Facebook AI Research, FAIR engineer Nikhila Ravi said. Working in 3D is important for rendering 3D objects or scenes that appear in mixed reality or virtual reality. It can also be used to tackle AI challenges like robotic grasping or helping autonomous vehicles understand the position of nearby objects.

Rendering a simple shape into a proper object with geometry, texture, and other material properties is a painstakingly long process; however, with AI, researchers can now do this rendering ten times faster than the real-time. A machine learning model is trained on images that are closer to the target. When it is presented with a shape and matching properties, it would recommend a photorealistic image. This opened a whole new field altogether -- differentiable programming. Traditional rendering engines are not differentiable, so they can't be incorporated into deep learning pipelines.

But research in 3D deep learning has been limited because of the lack of sufficient tools and resources to support the complexities of using neural networks with 3D data and the fact that many traditional graphic operators are not differentiable. Facebook AI has built and is now releasing PyTorch3D, a highly modular and optimized library with unique capabilities designed to make 3D deep learning easier with PyTorch. PyTorch3D provides a set of frequently used 3D operators and loss functions for 3D data that are fast and differentiable, as well as a modular differentiable rendering API -- enabling researchers to import these functions into current state-of-the-art deep learning systems right away. PyTorch3D was recently a catalyst in Facebook AI's work to build Mesh R-CNN, which achieved full 3D object reconstruction from images of complex interior spaces. We fused PyTorch3D with our highly optimized 2D recognition library, Detectron2, to successfully push object understanding to the third dimension.