Context is Environment

One key problem in AI research is to build systems that generalize across a wide range of test environments. In principle, these algorithms should discard spurious correlations present only in certain training environments, and capture invariant patterns appearing across conditions. For example, we would like to build self-driving systems that, while trained on data from environments with varying weather conditions, traffic conditions, and driving rules, can perform satisfactorily in completely new environments. Unfortunately, this has so far been a far cry: models trained catastrophically fail to generalize to unseen weather conditions [Lechner et al., 2022]. Despite its importance, how to perform well beyond the distribution of the training data remains a burning question. In fact, entire research groups are devoted to study generalization, major international conferences offer well-attended workshops dedicated to the issue [Wald et al., 2023], and news articles remind us of the profound societal impact from failures of ML systems [Angwin et al., 2016]. Research efforts have so far produced domain generalization algorithms that fall into one out of two broad categories. On the one hand, invariance proposals [Ganin et al., 2016, Peters et al., 2016, Arjovsky et al., 2019], illustrated in Figure 1a, discard all environment-specific information, thus removing excessive signal about the problem. On the other hand, marginal transfer proposals [Blanchard et al., 2011, Li et al., 2016, Zhang et al., 2020, Bao and Karaletsos, 2023], also illustrated in Figure 1b, summarize observed inputs in each environment as a coarse embedding, diluting important signal at the example level.