Performance Analysis
Uncertainty-Aware Tabular Prediction: Evaluating VBLL-Enhanced TabPFN in Safety-Critical Medical Data
Predictive models are being increasingly used across a wide range of domains, including safety-critical applications such as medical diagnosis and criminal justice. Reliable uncertainty estimation is a crucial task in such settings. Tabular Prior-data Fitted Network (TabPFN) is a recently proposed machine learning foundation model for tabular dataset, which uses a generative transformer architecture. Variational Bayesian Last Layers (VBLL) is a state-of-the-art lightweight variational formulation that effectively improves uncertainty estimation with minimal computational overhead. In this work we aim to evaluate the performance of VBLL integrated with the recently proposed TabPFN in uncertainty calibration. Our experiments, conducted on three benchmark medical tabular datasets, compare the performance of the original TabPFN and the VBLL-integrated version. Contrary to expectations, we observed that original TabPFN consistently outperforms VBLL integrated TabPFN in uncertainty calibration across all datasets.
Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees
Zeighami, Sepanta, Shankar, Shreya, Parameswaran, Aditya
Large Language Models (LLMs) are being increasingly used as a building block in data systems to process large text datasets. To do so, LLM model providers offer multiple LLMs with different sizes, spanning various cost-quality trade-offs when processing text at scale. Top-of-the-line LLMs (e.g., GPT-4o, Claude Sonnet) operate with high accuracy but are prohibitively expensive when processing many records. To avoid high costs, more affordable but lower quality LLMs (e.g., GPT-4o-mini, Claude Haiku) can be used to process records, but we need to ensure that the overall accuracy does not deviate substantially from that of the top-of-the-line LLMs. The model cascade framework provides a blueprint to manage this trade-off, by using the confidence of LLMs in their output (e.g., log-probabilities) to decide on which records to use the affordable LLM. However, existing solutions following this framework provide only marginal cost savings and weak theoretical guarantees because of poor estimation of the quality of the affordable LLM's outputs. We present BARGAIN, a method that judiciously uses affordable LLMs in data processing to significantly reduce cost while providing strong theoretical guarantees on the solution quality. BARGAIN employs a novel adaptive sampling strategy and statistical estimation procedure that uses data and task characteristics and builds on recent statistical tools to make accurate estimations with tight theoretical guarantees. Variants of BARGAIN can support guarantees on accuracy, precision, or recall of the output. Experimental results across 8 real-world datasets show that BARGAIN reduces cost, on average, by up to 86% more than state-of-the-art, while providing stronger theoretical guarantees on accuracy of output, with similar gains when guaranteeing a desired level of precision or recall.
A Dataset for Distilling Knowledge Priors from Literature for Therapeutic Design
Jones, Haydn Thomas, Maus, Natalie, Ludan, Josh Magnus, Huan, Maggie Ziyu, Liang, Jiaming, Torres, Marcelo Der Torossian, Liang, Jiatao, Ives, Zachary, Barash, Yoseph, de la Fuente-Nunez, Cesar, Gardner, Jacob R., Yatskar, Mark
AI-driven discovery can greatly reduce design time and enhance new therapeutics' effectiveness. Models using simulators explore broad design spaces but risk violating implicit constraints due to a lack of experimental priors. For example, in a new analysis we performed on a diverse set of models on the GuacaMol benchmark using supervised classifiers, over 60\% of molecules proposed had high probability of being mutagenic. In this work, we introduce Medex, a dataset of priors for design problems extracted from literature describing compounds used in lab settings. It is constructed with LLM pipelines for discovering therapeutic entities in relevant paragraphs and summarizing information in concise fair-use facts. Medex consists of 32.3 million pairs of natural language facts, and appropriate entity representations (i.e. SMILES or refseq IDs). To demonstrate the potential of the data, we train LLM, CLIP, and LLava architectures to reason jointly about text and design targets and evaluate on tasks from the Therapeutic Data Commons (TDC). Medex is highly effective for creating models with strong priors: in supervised prediction problems that use our data as pretraining, our best models with 15M learnable parameters outperform larger 2B TxGemma on both regression and classification TDC tasks, and perform comparably to 9B models on average. Models built with Medex can be used as constraints while optimizing for novel molecules in GuacaMol, resulting in proposals that are safer and nearly as effective. We release our dataset at https://huggingface.co/datasets/medexanon/Medex, and will provide expanded versions as available literature grows.
The Precautionary Principle and the Innovation Principle: Incompatible Guides for AI Innovation Governance?
In policy debates concerning the governance and regulation of Artificial Intelligence (AI), both the Precautionary Principle (PP) and the Innovation Principle (IP) are advocated by their respective interest groups. Do these principles offer wholly incompatible and contradictory guidance? Does one necessarily negate the other? I argue here that provided attention is restricted to weak-form PP and IP, the answer to both of these questions is "No." The essence of these weak formulations is the requirement to fully account for type-I error costs arising from erroneously preventing the innovation's diffusion through society (i.e. mistaken regulatory red-lighting) as well as the type-II error costs arising from erroneously allowing the innovation to diffuse through society (i.e. mistaken regulatory green-lighting). Within the Signal Detection Theory (SDT) model developed here, weak-PP red-light (weak-IP green-light) determinations are optimal for sufficiently small (large) ratios of expected type-I to type-II error costs. For intermediate expected cost ratios, an amber-light 'wait-and-monitor' policy is optimal. Regulatory sandbox instruments allow AI testing and experimentation to take place within a structured environment of limited duration and societal scale, whereby the expected cost ratio falls within the 'wait-and-monitor' range. Through sandboxing regulators and innovating firms learn more about the expected cost ratio, and what respective adaptations -- of regulation, of technical solution, of business model, or combination thereof, if any -- are needed to keep the ratio out of the weak-PP red-light zone. Nevertheless AI foundation models are ill-suited for regulatory sandboxing as their general-purpose nature precludes credible identification of misclassification costs.
Self-supervised Learning Of Visual Pose Estimation Without Pose Labels By Classifying LED States
Carlotti, Nicholas, Nava, Mirko, Giusti, Alessandro
We introduce a model for monocular RGB relative pose estimation of a ground robot that trains from scratch without pose labels nor prior knowledge about the robot's shape or appearance. At training time, we assume: (i) a robot fitted with multiple LEDs, whose states are independent and known at each frame; (ii) knowledge of the approximate viewing direction of each LED; and (iii) availability of a calibration image with a known target distance, to address the ambiguity of monocular depth estimation. Training data is collected by a pair of robots moving randomly without needing external infrastructure or human supervision. Our model trains on the task of predicting from an image the state of each LED on the robot. In doing so, it learns to predict the position of the robot in the image, its distance, and its relative bearing. At inference time, the state of the LEDs is unknown, can be arbitrary, and does not affect the pose estimation performance. Quantitative experiments indicate that our approach: is competitive with SoA approaches that require supervision from pose labels or a CAD model of the robot; generalizes to different domains; and handles multi-robot pose estimation.
Multi-pathology Chest X-ray Classification with Rejection Mechanisms
Aperstein, Yehudit, Tzahar, Amit, Gottlib, Alon, Verber, Tal, Damti, Ravit Shagan, Apartsin, Alexander
Overconfidence in deep learning models poses a significant risk in high - stakes medical imaging tasks, particularly in multi - label classification of chest X - rays, where multiple co - occurring pathologies must be detected simultaneously. This study introduces an uncertainty - aware framework for chest X - ray diagnosis based on a DenseNet - 121 backbone, enhanced with two selective prediction mechanisms: entropy - based rejection and confidence interval - based rejection. Both methods enable the model to abstain from un certain predictions, improving reliability by deferring ambiguous cases to clinical experts. A quantile - based calibration procedure is employed to tune rejection thresholds using either global or class - specific strategies. Experiments conducted on three la rge public datasets (PadChest, NIH ChestX - ray14, and MIMIC - CXR) demonstrate that selective rejection improves the trade - off between diagnostic accuracy and coverage, with entropy - based rejection yielding the highest average A U C across all pathologies. Thes e results support the integration of selective prediction into AI - assisted diagnostic workflows, providing a practical step toward safer, uncertainty - aware deployment of deep learning in clinical settings. Automating medical diagnosis with deep learning has shown great potential, particularly in medical imaging domains such as chest X - ray analysis. Convolutional neural networks, including architectures like DenseNet - 121, have demonstrated strong performance in detecting a range of thoracic pathologies [1],[2] . However, successfully integrating such models into clinical workflows requires more than high classification accuracy . I t demands robust mechanisms for managing uncertainty and ensuring patient safety. 2 Figure 1: Overview of the proposed selective chest X - ray classification framework .
Data-Driven Energy Estimation for Virtual Servers Using Combined System Metrics and Machine Learning
This paper presents a machine learning-based approach to estimate the energy consumption of virtual servers without access to physical power measurement interfaces. Using resource utilization metrics collected from guest virtual machines, we train a Gradient Boosting Regressor to predict energy consumption measured via RAPL on the host. We demonstrate, for the first time, guest-only resource-based energy estimation without privileged host access with experiments across diverse workloads, achieving high predictive accuracy and variance explained ($0.90 \leq R^2 \leq 0.97$), indicating the feasibility of guest-side energy estimation. This approach can enable energy-aware scheduling, cost optimization and physical host independent energy estimates in virtualized environments. Our approach addresses a critical gap in virtualized environments (e.g. cloud) where direct energy measurement is infeasible.
Detection of Anomalous Behavior in Robot Systems Based on Machine Learning
Nissan, Mahfuzul I., Aktar, Sharmin
Ensuring the safe and reliable operation of robotic systems is paramount to prevent potential disasters and safeguard human well-being. Despite rigorous design and engineering practices, these systems can still experience malfunctions, leading to safety risks. In this study, we present a machine learning-based approach for detecting anomalies in system logs to enhance the safety and reliability of robotic systems. We collected logs from two distinct scenarios using CoppeliaSim and comparatively evaluated several machine learning models, including Logistic Regression (LR), Support Vector Machine (SVM), and an Autoencoder. Our system was evaluated in a quadcopter context (Context 1) and a Pioneer robot context (Context 2). Results showed that while LR demonstrated superior performance in Context 1, the Autoencoder model proved to be the most effective in Context 2. This highlights that the optimal model choice is context-dependent, likely due to the varying complexity of anomalies across different robotic platforms. This research underscores the value of a comparative approach and demonstrates the particular strengths of autoencoders for detecting complex anomalies in robotic systems.
HGEN: Heterogeneous Graph Ensemble Networks
Shen, Jiajun, Jin, Yufei, He, Yi, Zhu, Xingquan
This paper presents HGEN that pioneers ensemble learning for heterogeneous graphs. We argue that the heterogeneity in node types, nodal features, and local neighborhood topology poses significant challenges for ensemble learning, particularly in accommodating diverse graph learners. Our HGEN framework ensembles multiple learners through a meta-path and transformation-based optimization pipeline to uplift classification accuracy. Specifically, HGEN uses meta-path combined with random dropping to create Allele Graph Neural Networks (GNNs), whereby the base graph learners are trained and aligned for later ensembling. To ensure effective ensemble learning, HGEN presents two key components: 1) a residual-attention mechanism to calibrate allele GNNs of different meta-paths, thereby enforcing node embeddings to focus on more informative graphs to improve base learner accuracy, and 2) a correlation-regularization term to enlarge the disparity among embedding matrices generated from different meta-paths, thereby enriching base learner diversity. We analyze the convergence of HGEN and attest its higher regularization magnitude over simple voting. Experiments on five heterogeneous networks validate that HGEN consistently outperforms its state-of-the-art competitors by substantial margin.
A Co-Training Semi-Supervised Framework Using Faster R-CNN and YOLO Networks for Object Detection in Densely Packed Retail Images
Yazdanjouei, Hossein, Mansouri, Arash, Shokouhifar, Mohammad
Abstract: This study proposes a semi-supervised co-training framework for object detection in densely packed retail environments, where limited labe led data and complex conditions pose major challenges. The framework combines Faster R-CNN (utilizing a ResNe t backbone) for precise localiza tion with YOLO (employing a Darknet backbone) for global context, enabling mutual pseudo-label exchange that impr oves accuracy in scenes with occlusion and overlapping objects. To strengthe n classification, it employs a n ensemble of XGBoost, Random Forest, and SVM, utilizing diverse feature representations for higher robustness . Hyperparameters are optimized using a metaheuristic-driven algorithm, enhancing precision and efficiency across mod els. By minimizing relianc e on manual labeling, the approach reduces annotation costs and adapts effectively to fre quent product and layout changes common in retail. Experiments on the SKU-110k datase t demonstrate strong performa nce, highlighting the scal ability and practicality of the proposed framework for real-world retail applications such as automated inventory tracking, product monitoring, and checkout systems. Keywords: Retail object detection; Densely packed scenes; Semi-supervised learning; Co-training method; Faster R-CNN; Metaheuristic optim ization; YOLO integration. Detecting objects in densely pack ed retail environments has bec ome essential due to the increasing demand for automation in inventory management, product recognition, and ef ficient checkout processes in modern retail.