Reconciling Individual Probability Forecasts

Probabilistic modelling in machine learning and statistics predicts "individual probabilities" as a matter of course. In weather forecasting, we speak of the probability of rain tomorrow; in life insurance underwriting we speak of the probability that Alice will die in the next 12 months; in recidivism prediction we speak of the probability that an inmate Bob will commit a violent crime within 18 months of being released on parole; in predictive medicine we speak of the probability that Carol will develop breast cancer before the age of 50 -- and so on. But these are not repeated events: we have no way of directly measuring an "individual probability" -- and indeed, even the semantics of an individual probability are unclear and have been the subject of deep interrogation within the philosophy of science and statistics [Hájek, 2007, Dawid, 2017] and theoretical computer science [Dwork et al., 2021]. Within the philosophy of science, puzzles related to individual probability have been closely identified with "the reference class problem" [Hájek, 2007]. This is a close cousin of a concern that has recently arisen in the context of fairness in machine learning called the "predictive multiplicity problem" (a focal subset of "model multiplicity problems") [Marx et al., 2020, Black et al., 2022] which Breiman [2001] earlier called the "Rashomon Effect". At the core of both of these problems is the fact that from a data sample that is much smaller than the data universe (i.e. the set of all possible observations), we will have observed at most one individual with a particular set of characteristics, and at most one outcome for the event that an"individual probability" speaks to: It will either rain tomorrow or it will not; Alice will either die within the next year or she will not; etc. We do not have the luxury of observing a large number of repetitions and taking averages. Dawid[2017] lays out two broad classes of perspectives on individual probabilities: the group to individual perspective and the individual to group perspective.