Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data