APLe: Token-Wise Adaptive for Multi-Modal Prompt Learning