Ask Your Distribution Shift if Pre-Training is Right for You