Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication