Firstly, these methods implicitly assume that all group combinations are represented during training. To illustrate this, we introduce a systematic generalization task on the MPI3D dataset and discover that current algorithms fail to improve the ERM baseline when combinations of observed attribute values are missing.

In online learning literature, stochastic and adversarial settings are two of the most well-studied cases.

Learning high-quality feature embeddings efficiently and effectively is critical for the performance of web-scale machine learning systems.

Whereas RR arbitrarily permutes training examples, GraB leverages stale gradients from prior epochs to order examples -- achieving a provably faster convergence rate than RR.