IW-GAE: Importance weighted group accuracy estimation for improved calibration and model selection in unsupervised domain adaptation

In this work, we consider a classification problem in unsupervised domain adaptation (UDA). UDA aims to transfer knowledge from a source domain with ample labeled data to enhance the performance in a target domain where labeled data is unavailable. In UDA, the source and target domains have different data generating distributions, so the core challenge is to transfer knowledge contained in the labeled dataset in the source domain to the target domain under the distribution shifts. Over the decades, significant improvements in the transferability from source to target domains have been made, resulting in areas like domain alignment (Ben-David et al., 2010; Ganin et al., 2016; Long et al., 2018; Zhang et al., 2019) and self-training (Cai et al., 2021; Chen et al., 2020; Liu et al., 2021). Improving calibration performance, which is about matching predictions regarding a random event to the long-term occurrence of the event (Dawid, 1982), is of central interest in the machine learning community due to its significance to safe and trustworthy deployment of machine learning models in critical real-world decision-making systems (Amodei et al., 2016; Lee and See, 2004). In independent and identically distributed (i.i.d.) settings, calibration performance has been significantly improved by various approaches (Gal and Ghahramani, 2016; Guo et al., 2017; Lakshminarayanan et al., 2017). However, producing well-calibrated predictions in UDA remains challenging due to the distribution shifts. Specifically, Wang et al. (2020) show the discernible compromise in calibration performance as an offset against the enhancement of target accuracy. A further observation reveals that state-of-the-art calibrated classifiers in the i.i.d.