Escaping Optimization Stagnation: Taking Steps Beyond Task Arithmetic via Difference Vectors