While the pace of development of AI has rapidly progressed in recent years, the implementation of safe and effective regulatory frameworks has lagged behind. In particular, the adaptive nature of AI models presents unique challenges to regulators as updating a model can improve its performance but also introduce safety risks. In the US, the Food and Drug Administration (FDA) has been a forerunner in regulating and approving hundreds of AI medical devices. To better understand how AI is updated and its regulatory considerations, we systematically analyze the frequency and nature of updates in FDA-approved AI medical devices. We find that less than 2% of all devices report having been updated by being re-trained on new data. Meanwhile, nearly a quarter of devices report updates in the form of new functionality and marketing claims. As an illustrative case study, we analyze pneumothorax detection models and find that while model performance can degrade by as much as 0.18 AUC when evaluated on new sites, re-training on site-specific data can mitigate this performance drop, recovering up to 0.23 AUC. However, we also observed significant degradation on the original site after re-training using data from new sites, providing insight from one example that challenges the current one-model-fits-all approach to regulatory approvals. Our analysis provides an in-depth look at the current state of FDA-approved AI device updates and insights for future regulatory policies toward model updating and adaptive AI.

Today, the U.S. Food and Drug Administration authorized marketing of the GI Genius, the first device that uses artificial intelligence (AI) based on machine learning to assist clinicians in detecting lesions (such as polyps or suspected tumors) in the colon in real time during a colonoscopy. "Artificial intelligence has the potential to transform health care to better assist health care providers and improve patient care. When AI is combined with traditional screenings or surveillance methods, it could help find problems early on, when they may be easier to treat," said Courtney H. Lias, Ph.D. acting director of the GastroRenal, ObGyn, General Hospital and Urology Devices Office in the FDA's Center for Devices and Radiological Health. "Studies show that during colorectal cancer screenings, missed lesions can be a problem even for well-trained clinicians. With the FDA's authorization of this device today, clinicians now have a tool that could help improve their ability to detect gastrointestinal lesions they may have missed otherwise."

The U.S. Food and Drug Administration (FDA) has just approved an AI-based diagnostic tool that can accurately detect wrist fractures. Imagene's OsteoDetect uses machine learning algorithms to study 2D X-rays for the signs of wrist fractures. "Artificial intelligence algorithms have tremendous potential to help health care providers diagnose and treat medical conditions," said Robert Ochs, Ph.D., acting deputy director for radiological health, Office of In Vitro Diagnostics and Radiological Health in the FDA's Center for Devices and Radiological Health. "This software can help providers detect wrist fractures more quickly and aid in the diagnosis of fractures." OsteoDetect isn't about to replace doctors but it can help improve fracture detection and get the correct diagnosis and treatment quickly.