Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild