Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program

This paper presents a unified framework to translate between pairs of shape abstractions: Text Point Cloud Program. We propose Neural Shape Compiler to model the abstraction transformation as a conditional generation process. It converts 3D shapes of three abstract types into discrete shape code, transforms each shape code into code of other abstract types through the proposed ShapeCode Transformer, and decodes them to output the target shape abstraction. Point Cloud code is obtained in a class-agnostic way by the proposed PointVQVAE. On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in Text = Point Cloud, Point Cloud = Text, Point Cloud = Program, and Point Cloud Completion tasks.