Policy Brief

As the development and adoption of AI-enabled healthcare continue to accelerate, regulators and researchers are beginning to confront oversight concerns in the clinical evaluation process that could yield negative consequences on patient health if left unchecked. Since 2015, the United States Food and Drug Administration (FDA) has evaluated and granted clearance for over 100 AI-based medical devices using a fairly rudimentary evaluation process that is in dire need of improvement as these evaluations have not been adapted to address the unique concerns surrounding AI. This brief examined this evaluation process and analyzed how devices were evaluated before approval. We analyzed public records for all 130 FDA-approved medical AI devices between January 2015 and December 2020 and observed significant variety and limitations in test-data rigor and what developers considered appropriate clinical evaluation. When we performed an analysis of a well-established diagnostic task (pneumothorax, or collapsed lung) using three sets of training data, the level of error exhibited between white and Black patients increased dramatically.