Evaluating the Adversarial Robustness of Retrieval-Based In-Context Learning for Large Language Models