Is Parameter Collision Hindering Continual Learning in LLMs?