The Value of Information in Human-AI Decision-making

As the performance of artificial intelligence (AI) models improves, workflows in which human and AI model-based judgments are combined to make decisions are sought in medicine, finance, and other domains. Though statistical models often make more accurate predictions than human experts on average [Ægisdóttir et al., 2006, Grove et al., 2000, Meehl, 1954], whenever humans have access to additional information over the AI, there is potential to achieve complementary performance by pairing the two, i.e., better performance than either the human or AI alone. For example, a physician may have access to additional information that may not be captured in tabular electronic health records or other structured data [Alur et al., 2024b]. However, evidence of complementary performance between humans and AI is limited, with many studies showing that human-AI teams underperform an AI alone [Buçinca et al., 2020, Bussone et al., 2015, Green and Chen, 2019, Jacobs et al., 2021, Lai and Tan, 2019, Vaccaro and Waldo, 2019, Kononenko, 2001]. A solid understanding of such results is limited by the fact that most analyses of human-AI decision-making focus on ranking the performance of human-AI teams or each individually using measures like posthoc decision accuracy.