Changes in the data distribution at test time can have deleterious effects on the performance of predictive models $p(y|x)$.We consider situations where there are additional meta-data labels (such as group labels), denoted by $z$, that can account for such changes in the distribution.In particular, we assume that the prior distribution $p(y,z)$, which models the dependence between the class label $y$ and the nuisance factors $z$, may change across domains, either due to a change in the correlation between these terms, or a change in one of their marginals.However, we assume that the generative model for features $p(x|y,z)$ is invariant across domains.We note that this corresponds to an expanded version of the widely used label shift assumption, where the labels now also include the nuisance factors $z$.

Changes in the data distribution at test time can have deleterious effects on the performance of predictive models p(y x) .We consider situations where there are additional meta-data labels (such as group labels), denoted by z, that can account for such changes in the distribution.In particular, we assume that the prior distribution p(y,z), which models the dependence between the class label y and the "nuisance" factors z, may change across domains, either due to a change in the correlation between these terms, or a change in one of their marginals.However, we assume that the generative model for features p(x y,z) is invariant across domains.We note that this corresponds to an expanded version of the widely used "label shift" assumption, where the labels now also include the nuisance factors z .

Image classifiers often use spurious patterns, such as "relying on the presence of a person to detect a tennis racket, which do not generalize. In this work, we present an end-to-end pipeline for identifying and mitigating spurious patterns for such models, under the assumption that we have access to pixel-wise object-annotations. We start by identifying patterns such as "the model's prediction for tennis racket changes 63% of the time if we hide the people." Then, if a pattern is spurious, we mitigate it via a novel form of data augmentation. We demonstrate that our method identifies a diverse set of spurious patterns and that it mitigates them by producing a model that is both more accurate on a distribution where the spurious pattern is not helpful and more robust to distribution shift.