Black-box Optimization of LLM Outputs by Asking for Directions

We present a novel approach for attacking black-box large language models (LLMs) by exploiting their ability to express confidence in natural language. Existing black-box attacks require either access to continuous model outputs like log-its or confidence scores (which are rarely available in practice), or rely on proxy signals from other models. Instead, we demonstrate how to prompt LLMs to express their internal confidence in a way that is sufficiently calibrated to enable effective adversarial optimization. We apply our general method to three attack scenarios: adversarial examples for vision-LLMs, jailbreaks and prompt injections. Our attacks successfully generate malicious inputs against systems that only expose textual outputs, thereby dramatically expanding the attack surface for deployed LLMs. We further find that better and larger models exhibit superior calibration when expressing confidence, creating a concerning security paradox where model capability improvements directly enhance vulnerability.Figure 1: An illustration of our optimization for adversarial examples for vision-LLMs. Moreover, our general method can be applied to many applications, including jailbreaks and prompt injections. Large language models (LLMs) have become integral components of countless applications, from chatbots and code assistants to autonomous agents and content generation systems. However, this widespread deployment has also created new attack surfaces. Adversaries can manipulate inputs to these systems in various ways: designing jailbreak prompts that bypass safety guardrails to elicit harmful content (Zou et al., 2023), injecting malicious instructions into data processed by LLMpowered agents (Willison, 2022; Goodside, 2022), or crafting adversarial examples that cause vision large language models (vision-LLMs) to misclassify images (Li et al., 2025; Hu et al., 2025). The difficulty in mounting these attacks depends on the adversary's access level. In white-box settings, where attackers have full access to model parameters and gradients, adversarial optimization is straightforward--one can simply perform gradient descent to optimize inputs for a desired output (Shin et al., 2020; Zou et al., 2023). More commonly, attackers face black-box scenarios in which they can only query the model through an API.