Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation