A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias