Fine-Tuning Linear Layers Only Is a Simple yet Effective Way for Task Arithmetic