Learningto Modulate pre-trained Models in RL