Multimodal artificial intelligence (AI) systems have the potential to enhance clinical decision-making by interpreting various types of medical data. However, the effectiveness of these models across all medical fields is uncertain. Each discipline presents unique challenges that need to be addressed for optimal performance. This complexity is further increased when attempting to integrate different fields into a single model. Here, we introduce an alternative approach to multimodal medical AI that utilizes the generalist capabilities of a large language model (LLM) as a central reasoning engine. This engine autonomously coordinates and deploys a set of specialized medical AI tools. These tools include text, radiology and histopathology image interpretation, genomic data processing, web searches, and document retrieval from medical guidelines. We validate our system across a series of clinical oncology scenarios that closely resemble typical patient care workflows. We show that the system has a high capability in employing appropriate tools (97%), drawing correct conclusions (93.6%), and providing complete (94%), and helpful (89.2%) recommendations for individual patient cases while consistently referencing relevant literature (82.5%) upon instruction. This work provides evidence that LLMs can effectively plan and execute domain-specific models to retrieve or synthesize new information when used as autonomous agents. This enables them to function as specialist, patient-tailored clinical assistants. It also simplifies regulatory compliance by allowing each component tool to be individually validated and approved. We believe, that our work can serve as a proof-of-concept for more advanced LLM-agents in the medical domain.

Innovations in cancer research through interdisciplinary team science approaches will help shape the future of patient care. Integration and mining of health care data from various sources will allow researchers to gain more insights into cancer biology and thereby improve patient outcomes. Cutting-edge technologies that fuel the full spectrum of cancer science from bench to bedside will accelerate the pace at which we increase our understanding of cancer biology while transforming the future of clinical practice. This is an exciting era of cancer research. Approval of novel therapeutics, coupled with an increasing public awareness of cancer prevention and early detection, has led to dramatic reductions in overall cancer mortality rates for all Americans. Recent discoveries in the fields of cancer genomics and immunology have firmly established two new pillars of cancer care--molecularly targeted therapy and immunotherapy--which are benefiting many patients with a wide range of cancer types.