When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Models