When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations