Adaptive Knowledge Distillation between Text and Speech Pre-trained Models