Think Small, Act Big: Primitive Prompt Learning for Lifelong Robot Manipulation