Auditing the Fairness of COVID-19 Forecast Hub Case Prediction Models

The COVID-19 Forecast Hub was founded in 2020 and serves as a "central repository of COVID-19 forecasts from over 50 independent research groups" [1]. Participant research groups submit county, state and national US COVID-19 forecasts with a standardized format; and the Forecast Hub provides an interactive visualization tool to help decision makers and the general public analyze weekly predictions for COVID-19 hospitalizations, cases and deaths. The standardized predictions collected from all research groups, as well as the predictions for an ensemble model that brings all individual predictions together, are also shared with the Centers for Disease Control and Prevention (CDC) who uses these results for their official COVID-19 communications [2]. The COVID-19 Forecast Hub has been, and continues to be, a critical centralized resource to promote transparent decision making. Nevertheless, by focusing exclusively on prediction accuracy at different spatial granularities (e.g., county or state), the Forecast Hub fails to evaluate whether the proposed models are fair i.e., share similar prediction performance across social determinants that have been known to play a role in COVID-19 including race, ethnicity and rurality [3, 4]. Diverse prediction performance across social determinants - for example, higher prediction errors for a given minority race or ethnicity - could negatively impact resource allocation and intervention decisions e.g., hospital beds or stay-at-home orders, given that the CDC appears to be using the Forecast Hub predictions for official communications that subsequently inform policy decisions [2]. In other words, allocation or intervention harms might occur if models from the Forecast Hub are used to inform decision making across communities without taking into account fairness metrics [5]. There are many reasons why the COVID-19 prediction performance can be different across social determinants such as race, ethnicity or urbanization levels. The Forecast Hub's COVID-19 prediction models are trained on datasets containing COVID-19