TAPO: Task-Referenced Adaptation for Prompt Optimization