It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition