The Realignment Problem: When Right becomes Wrong in LLMs