Credal Two-Sample Tests of Epistemic Ignorance

Science is inherently inductive and thus involves uncertainties. They are commonly categorized as aleatoric uncertainty (AU), which refers to inherent variability, and epistemic uncertainty (EU), arising from limited information such as finite data or model assumptions (Hora, 1996). These uncertainties often overlap, as scientists may be epistemically uncertain about the aleatoric variation in their inquiry. Distinguishing and acknowledging them is crucial for the safe and trustworthy deployment of intelligent systems (Kendall and Gal, 2017; Hüllermeier and Waegeman, 2021), as they lead to different down-stream decisions. For example, experimental design aims to reduce EU (Nguyen et al., 2019; Chau et al., 2021b; Adachi et al., 2024), while risk management uses hedging strategy to address AU (Mashrur et al., 2020) While AU is often modelled using probability distributions, modelling EU--particularly in states of epistemic ignorance, also known as partial ignorance or incomplete knowledge (Dubois et al., 1996)--poses greater challenges. For instance, a scientist analysing insulin levels in Germany may have data from multiple hospitals, each representing aleatoric variation as a probability distribution. However, these distributions are merely proxies for the population-level insulin distribution, which is difficult to infer due to data collection limitations. A Bayesian approach could aggregate the data based on a prior if the representativeness of each source is known, but in many cases, scientists operate under partial ignorance, lacking such prior information (Bromberger, 1971). Assigning a uniform prior by following the principle of indifference (Keynes, 1921) and maximum entropy principle (Jaynes, 1957), or applying Jeffrey's prior by following the principle of transformation groups (Jaynes, 1968) only reflects indifference, not epistemic ignorance.