Therefore, estimating how well a given model might perform on the new data is an important step toward reliable ML applications. This isverychallenging, however,asthedata distribution can change inflexible ways, and we may not haveanylabels on the new data, which is often the case in monitoring settings. In this paper, we propose a new distribution shift model, Sparse Joint Shift (SJS), which considers the joint shift of both labels and afew features.

In order to accelerate the NES search phase, we generated the pool using the weight sharing schemes proposed by Random Search with WeightSharing[37]andDARTS[39]. Specifically, we trained one-shot weight-sharing models usingeachof these two algorithms, then we sampled architectures from the weightshared models uniformly at random to build the pool.

In this section, we follow the same notations described in Section 3.1 of the main paper.