Medical Interpretability and Knowledge Maps of Large Language Models

We present a systematic study of medical-domain interpretability in Large Language Models (LLMs). We study how the LLMs both represent and process medical knowledge through four different interpretability techniques: (1) UMAP projections of intermediate activations, (2) gradient-based saliency with respect to the model weights, (3) layer lesioning/removal and (4) activation patching. We present knowledge maps of five LLMs which show, at a coarse-resolution, where knowledge about patient's ages, medical symptoms, diseases and drugs is stored in the models. In addition, we find several interesting phenomena: (i) age is often encoded in a non-linear and sometimes discontinuous manner at intermediate layers in the models, (ii) the disease progression representation is non-monotonic and circular at certain layers of the model, (iii) in Llama3.3-70B, These results can guide future research on fine-tuning, unlearning or de-biasing LLMs for medical tasks by suggesting at which layers in the model these techniques should be applied. We attached our source code to the paper for reproducibility. Large Language Models (LLMs) have achieved impressive performance on a variety of tasks, including for coding, reasoning and in-context learning Radford et al. (2019); Coignion et al. (2024); Hao et al. (2024); Wang et al. (2023); Szymanski et al. (2025); Nguyen et al. (2024). However, the understanding of the underlying mechanisms of how such models represent and store knowledge is still scarce. This is particularly important for medical tasks, where insights about the LLMs' representations of patient demographics, diseases and drug treatments can help uncover hidden biases, thus making such insights crucial for building safe and trustworthy models. Over the past few years, a variety of mechanistic interpretability studies have been done on LLMs.