A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification