Test-time Adaptation for Regression by Subspace Alignment

This paper investigates test-time adaptation (TTA) for regression, where a regression model pre-trained in a source domain is adapted to an unknown target distribution with unlabeled target data. Although regression is one of the fundamental tasks in machine learning, most of the existing TT A methods have classification-specific designs, which assume that models output class-categorical predictions, whereas regression models typically output only single scalar values. To enable TT A for regression, we adopt a feature alignment approach, which aligns the feature distributions between the source and target domains to mitigate the domain gap. However, we found that naive feature alignment employed in existing TT A methods for classification is ineffective or even worse for regression because the features are distributed in a small subspace and many of the raw feature dimensions have little significance to the output. For an effective feature alignment in TT A for regression, we propose Significant-subspace Alignment (SSA) . SSA consists of two components: subspace detection and dimension weighting. Subspace detection finds the feature subspace that is representative and significant to the output. Then, the feature alignment is performed in the subspace during TT A. Meanwhile, dimension weighting raises the importance of the dimensions of the feature subspace that have greater significance to the output. We experimentally show that SSA outperforms various baselines on real-world datasets. Deep neural networks have achieved remarkable success in various tasks (LeCun et al., 1998a; Krizhevsky et al., 2012; He et al., 2016; Dargan et al., 2020). In particular, regression, which is one of the fundamental tasks in machine learning, is widely used in practical tasks such as human pose estimation or age prediction (Lathuili ` ere et al., 2019). The successes of deep learning have usually relied on the assumption that the training and test datasets are sampled from an i.i.d. In the real world, however, such an assumption is often invalid since the test data are sampled from distributions different from the training one due to distribution shifts caused by changes in environments. The performance of these models thus deteriorates when a distribution shift occurs (Hendrycks & Dietterich, 2019; Recht et al., 2019). To address this problem, test-time adaptation (TTA) (Liang et al., 2023) has been studied. TT A aims at adapting a model pre-trained in a source domain (training environment) to the target domain (test environment) with only unlabeled target data. However, most of the existing TT A methods are designed for classification; that is, TT A for regression has not been explored much (Liang et al., 2023). Regarding TT A for classification, two main approaches have been explored: entropy minimization and feature alignment.