Parameter Efficient Multi-task Model Fusion with Partial Linearization