One of the fundamental cognitive abilities of humans is to quickly resolve uncertainty by generating hypotheses and testing them via active trials.

We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider both epistemic and aleatoric uncertainties, where the former comes from the lack of knowledge about the ground truth (such as about facts or the language), and the latter comes from irreducible randomness (such as multiple possible answers).

Tweedie's formula, we show that we can perform gradient steps to sequentially optimize these objectives.

Most applications of machine learning to classification assume a closed set of balanced classes.

Interacting systems are prevalent in nature.

BACON allows computational resources to be focused on simulations that provide the most value in terms of reducing epistemic uncertainty.