Combining Diverse Feature Priors

The driving force behind deep learning's success is its ability to automatically discover predictive features in complex high-dimensional datasets. These features can generalize beyond the specific task at hand, thus enabling models to transfer to other (similar) tasks [DJV+14]. At the same time, the set of features that the model learns has a large impact on the model's performance on unseen inputs, especially in the presence of distribution shift [PBE+06; TE11; SKH+20] or spurious correlations [HM17; BVP18; Mei18]. Motivated by this, recent work focuses on encouraging specific modes of behavior by preventing the models from relying on certain features.