Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately