Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models