Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model