Towards General-Purpose Speech Abilities for Large Language Models Using Unpaired Data