Contrastive Embeddings for Neural Architectures

The performance of algorithms for neural architecture search strongly depends on the parametrization of the search space. We use contrastive learning to identify networks across different initializations based on their data Jacobians, and automatically produce the first architecture embeddings independent from the parametrization of the search space. Using our contrastive embeddings, we show that traditional black-box optimization algorithms, without modification, can reach state-of-the-art performance in Neural Architecture Search. As our method provides a unified embedding space, we perform for the first time transfer learning between search spaces. Finally, we show the evolution of embeddings during training, motivating future studies into using embeddings at different training stages to gain a deeper understanding of the networks in a search space. Traditionally, the design of state-of-the-art neural network architectures is informed by domain knowledge and requires a large amount of manual work to find the best hyperparameters. However, automated architecture search methods have recently achieved state-of-the-art results on tasks such as image classification, object detection, semantic segmentation and speech recognition, or even data augmentation and platform-aware optimization (Ren et al., 2020).