direction 0
MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports
Kyung, Sunggu, Park, Hyungbin, Seo, Jinyoung, Sung, Jimin, Kim, Jihyun, Kim, Dongyeong, Jo, Wooyoung, Nam, Yoojin, Park, Sangah, Kwon, Taehee, Lee, Sang Min, Kim, Namkug
Computed T omography (CT) plays a crucial role in clinical diagnosis, but the growing demand for CT examinations has raised concerns about diagnostic errors. While Multimodal Large Language Models (MLLMs) demonstrate promising comprehension of medical knowledge, their tendency to produce inaccurate information highlights the need for rigorous validation. However, existing medical visual question answering (VQA) benchmarks primarily focus on simple visual recognition tasks, lacking clinical relevance and failing to assess expert-level knowledge. W e introduce MedErr-CT, a novel benchmark for evaluating medical MLLMs' ability to identify and correct errors in CT reports through a VQA framework. The benchmark includes six error categories--four vision-centric errors (Omission, Insertion, Direction, Size) and two lexical error types (Unit, Typo)--and is organized into three task levels: classification, detection, and correction. Using this benchmark, we quantitatively assess the performance of state-of-the-art 3D medical MLLMs, revealing substantial variation in their capabilities across different error types. Our benchmark contributes to the development of more reliable and clinically applicable MLLMs, ultimately helping reduce diagnostic errors and improve accuracy in clinical practice.
Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?
Ilyas, Andrew, Engstrom, Logan, Santurkar, Shibani, Tsipras, Dimitris, Janoos, Firdaus, Rudolph, Larry, Madry, Aleksander
Deep reinforcement learning (RL) is at the core of some of the most publicized achievements of modern machine learning [19, 9, 1, 10]. To many, this framework embodies the promise of the real-world impact of machine learning. However, the deep RL toolkit has not yet attained the same level of engineering stability as, for example, the current deep (supervised) learning framework. Indeed, recent studies [3] demonstrate that state-of-the-art deep RL algorithms suffer from oversensitivity to hyperparameter choices, lack of consistency, and poor reproducibility. This state of affairs suggests that it might be necessary to reexamine the conceptual underpinnings of deep RL methodology. More precisely, the overarching question that motivates this work is: To what degree does the current practice of deep RL reflect the principles that informed its development? The specific focus of this paper is on deep policy gradient methods, a widely used class of deep RL algorithms. Our goal is to explore the extent to which state-of-the-art implementations of these methods succeed at realizing the key primitives of the general policy gradient framework.
Dataiku's Solution to SPHERE's Activity Recognition Challenge
Voisin, Maxime, Dreyfus-Schmidt, Leo, Gutierrez, Pierre, Ronsin, Samuel, Beillevaire, Marc
Our team won the second prize of the Safe Aging with SPHERE Challenge organized by SPHERE, in conjunction with ECML-PKDD and Driven Data. The goal of the competition was to recognize activities performed by humans, using sensor data. This paper presents our solution. It is based on a rich pre-processing and state of the art machine learning methods. From the raw train data, we generate a synthetic train set with the same statistical characteristics as the test set. We then perform feature engineering. The machine learning modeling part is based on stacking weak learners through a grid searched XGBoost algorithm. Finally, we use post-processing to smooth our predictions over time.