Reviews: Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

The paper presents a differentiable renderer (DIB-Render) that can render a coloured 3D mesh onto a 2D image. Having such renderer allows, for example, to train a neural network that can reconstruct a 3D shape of an object from a single image and render the shape onto a number of 2D views using different camera configurations. The learning can then be supervised by computing a reconstruction error between the computed rendering of a 3D shape and an actual image (using an L1 loss for the coloured image or Intersection over Union (IoU) for the binary silhouettes). The renderer is largely based on the soft rasterizer (Soft-Ras) proposed in [18, 19]. Unlike traditional non-differentiable rasterizers, which assign a binary score of whether a pixel in the image plane is covered by a triangle or not, Soft-Ras computes a soft score based on a distance of a pixel to the triangle (with an exponential or a sigmoid function of distance).