Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning

Figure 1: Within this work, we differentiate between various levels of image understanding: The original image (first row, left) can be interpreted as a panoramic semantic map (second row, left) by assigning a label to each pixel without differentiating between different instances of countable objects. Instances of countable objects are distinguished in the panoramic instance understanding (second row, right). The panoramic panoptic understanding (first row, right), which is the proposed method in this paper, builds on top of the previous understandings by eliminating their shortcomings: If possible different instances are distinguished and we guarantee that a label is assigned to each pixel. Abstract--In this work, we introduce panoramic panoptic combining supervised and contrastive training. A complete surrounding understanding provides a maximum of information to a mobile agent. ANOPTIC segmentation is the so far most complete segmentation task to describe the context of an image [1]. The domain shift from pinhole-to panoramic images is no exception. These properties have not been observed by the model during the training and make their correct segmentation Field of View challenging. Feature (PRF) framework which allows us to generate robust backbones via a contrastive pretext task. This poses severe problems due to the lack of does not only encourage similar features to be represented information containing the entire surrounding and the inability in a similar manner but more important, it pushes dissimilar of the agent to make proper decisions which may even lead features away from each other [16], [17]. This leads to well to accidents [5]. Thus, both pieces of information are equally separated clusters in the latent space of the backbone which important: the image should cover the entire surrounding and proves to mitigate distribution shift performance drops.