Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models