A Closer Look at In-Context Learning under Distribution Shifts