Convolutional Neural Networks With Heterogeneous Metadata

In autonomous driving, convolutional neural networks are the go-to tool for various perception tasks. Although CNNs are great at distilling information from camera images (or a sequence of them in form of a video clip), I constantly bump into all kinds of metadata that do not lend themselves to convolutional neural networks. Metadata, by traditional definition, means a set of data used to describe other data. All these properties make it hard for CNN to consume the metadata directly as CNN assumes a data representation on a regular-spaced grid, and neighboring data on the grid has a closer spatial or semantic relationship as well. One special case is lidar point cloud data.