Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding