DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks

LLM-integrated applications and agents-such as Bing Copilot [1], Google search with AI overviews [2], and Amazon's review highlights [3]-are emerging applications built upon large language models (LLMs). The growing popularity of LLM-integrated applications has led to the emergence of app stores, such as OpenAI's GPT Store and Poe [4], where developers can publish their LLMintegrated applications and users can access them, much like the Google Play and App Store for mobile apps. In general, an LLM-integrated application intends to perform a task (referred to as target task), such as webpage summarization in AI-assisted search. Towards this goal, an LLM-integrated application takes a prompt, which is the concatenation of an instruction (referred to as target instruction) and data (referred to as target data), as an input to query the backend LLM, whose response would solve the target task. The target instruction is often designed by an application developer to direct the backend LLM to perform the target task, while the data is the information to be processed by the backend LLM and is usually from an external source, e.g., the Internet. For instance, when the target task is webpage summarization in AI-assisted search, the target instruction can be "Please summarize the following web pages: [Text from relevant web pages].",