Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks