Modern vision models are trained on very large noisy datasets.

In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling.

In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior.