Pr$εε$mpt: Sanitizing Sensitive Prompts for LLMs

The recent advent of large language models (LLMs) have brought forth a fresh set of challenges for protecting users' data privacy. LLMs and their APIs present significant privacy concerns at inference time, which are fundamentally distinct from the well-documented risks of training data memorization [19, 50, 64, 98]. While the potential adversary in training data scenarios could be any API user, the threat during inference primarily stems from the model owner--typically the organization hosting the LLM. This inference stage poses a significant privacy risk, as prompts in natural language may include various types of sensitive information, from personally identifiable data like SSNs or credit card numbers to personal health or financial details. The ensuing privacy threat is exacerbated with the growing use of in-context learning, that involves presenting the LLM with a few training examples as part of the prompt during inference [17].