The Silent Majority: Demystifying Memorization Effect in the Presence of Spurious Correlations