NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering

Test-Time Adaptation (TT A) methods are often computationally expensive, require a large amount of data for effective adaptation, or are brittle to hyperparam-eters. Based on a theoretical foundation of the geometry of the latent space, we are able to significantly improve the alignment between source and distribution-shifted samples by re-centering target data embeddings at the origin. This insight motivates NEO - a hyperparameter-free fully TT A method, that adds no significant compute compared to vanilla inference. NEO is able to improve the classification accuracy of ViT -Base on ImageNet-C from 55.6% to 59.2% after adapting on just one batch of 64 samples. When adapting on 512 samples NEO beats all 7 TT A methods we compare against on ImageNet-C, ImageNet-R and ImageNet-S and beats 6/7 on CIFAR-10-C, while using the least amount of compute. NEO performs well on model calibration metrics and additionally is able to adapt from 1 class to improve accuracy on 999 other classes in ImageNet-C. On Raspberry Pi and Jetson Orin Nano devices, NEO reduces inference time by 63% and memory usage by 9% compared to baselines. Our results based on 3 ViT architectures and 4 datasets show that NEO can be used efficiently and effectively for TT A. A central challenge in machine learning is maintaining performance under distribution shifts between training and deployment. For instance, an image classifier may excel on curated training data but degrade on real-world inputs with snow, fog, or motion blur. Test-Time Adaptation (TT A) methods (Li et al., 2018; Wang et al., 2024; Liang et al., 2020; Wang et al., 2021; Niu et al., 2023) address this by leveraging unlabeled test samples without requiring access to training data, making them particularly suited to the modern setting of large pre-trained models.