Self-supervised Latent Space Optimization with Nebula Variational Coding

Abstract--Deep learning approaches process data in a layer-by-layer way with intermediate (or latent) features. We aim at designing a general solution to optimize the latent manifolds to improve the performance on classification, segmentation, completion and/or reconstruction through probabilistic models. This paper proposes a variational inference model which leads to a clustered embedding. We introduce additional variables in the latent space, called nebula anchors, that guide the latent variables to form clusters during training. T o prevent the anchors from clustering among themselves, we employ the variational constraint that enforces the latent features within an anchor to form a Gaussian distribution, resulting in a generative model we refer as Nebula Variational Coding (NVC). Since each latent feature can be labeled with the closest anchor, we also propose to apply metric learning in a self-supervised way to make the separation between clusters more explicit. As a consequence, the latent variables of our variational coder form clusters which adapt to the generated semantic of the training data, e.g. the categorical labels of each sample. We demonstrate experimentally that it can be used within different architectures designed to solve different problems including text sequence, images, 3D point clouds and volumetric data, validating the advantage of our proposed method. They rely on compressing the input to its latent representations while identifying its discriminative information. On the other hand, solutions in image processing [6], [7], [8], [9] and 3D understanding [8], [10], [11], [12], [13], [14] aim at preserving the information from the input data as part of the output, formulating additional skip connection [8], [9], [10], [12] and fusion [11] between them. Although there are traditional methods that can reduce the dimensionality of the input in an unsupervised way, such as principal component analysis (PCA) [15] and independent component analysis (ICA) [16], as well as in a supervised way, e.g.