The Price of Intelligence

The large number of ways to phrase a statement in natural language, combined with the core trained imperative to continue text the way a human would, means that nuances of human vulnerability to error and misinterpretation are also reproduced by these models. Hallucination, the tendency of LLMs to generate content that is factually incorrect or nonsensical. For example, a model might recall a fact from its training data or from its prompt with 99% probability (taken over the distribution of the decoding process) but miserably fail to recall it 1% of the time. Or, ignoring for a moment the stochasticity of decoding, it might recall the fact for 99% of the plausible prompts asking to do it but not for the remaining 1%. Indirect prompt injection, the potential for malicious instructions to be embedded within input data not under the user's direct control (such as emails), potentially altering the model's behavior in unexpected ways.