Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models