Zero-shot generalization across architectures for visual classification

Generalization to unseen data is a key desideratum for deep networks, but its relation to classification accuracy is unclear. Using a minimalist vision dataset and a measure of generalizability, we show that popular networks, from deep convolutional networks (CNNs) to transformers, vary in their power to extrapolate to unseen classes both across layers and across architectures. Accuracy is not a good predictor of generalizability, and generalization varies non-monotonically with layer depth. In deep learning for classification, generalization is typically considered in the context of training versus test sets (Zhang et al., 2021), where both contain examples from the same set of classes, and is measured via test set accuracy. Our goal is to investigate the capacity of a network to generalize its classification power to similar classes absent in the training set (unseen).