Differentiable Prompt Learning for Vision Language Models