Performance Analysis
Assisting Clinical Decisions for Scarcely Available Treatment via Disentangled Latent Representation
Xue, Bing, Said, Ahmed Sameh, Xu, Ziqi, Liu, Hanyang, Shah, Neel, Yang, Hanqing, Payne, Philip, Lu, Chenyang
Extracorporeal membrane oxygenation (ECMO) is an essential life-supporting modality for COVID-19 patients who are refractory to conventional therapies. However, the proper treatment decision has been the subject of significant debate and it remains controversial about who benefits from this scarcely available and technically complex treatment option. To support clinical decisions, it is a critical need to predict the treatment need and the potential treatment and no-treatment responses. Targeting this clinical challenge, we propose Treatment Variational AutoEncoder (TVAE), a novel approach for individualized treatment analysis. TVAE is specifically designed to address the modeling challenges like ECMO with strong treatment selection bias and scarce treatment cases. TVAE conceptualizes the treatment decision as a multi-scale problem. We model a patient's potential treatment assignment and the factual and counterfactual outcomes as part of their intrinsic characteristics that can be represented by a deep latent variable model. The factual and counterfactual prediction errors are alleviated via a reconstruction regularization scheme together with semi-supervision, and the selection bias and the scarcity of treatment cases are mitigated by the disentangled and distribution-matched latent space and the label-balancing generative strategy. We evaluate TVAE on two real-world COVID-19 datasets: an international dataset collected from 1651 hospitals across 63 countries, and a institutional dataset collected from 15 hospitals. The results show that TVAE outperforms state-of-the-art treatment effect models in predicting both the propensity scores and factual outcomes on heterogeneous COVID-19 datasets. Additional experiments also show TVAE outperforms the best existing models in individual treatment effect estimation on the synthesized IHDP benchmark dataset.
PseudoCell: Hard Negative Mining as Pseudo Labeling for Deep Learning-Based Centroblast Cell Detection
Seesawad, Narongrid, Ittichaiwong, Piyalitt, Sudhawiyangkul, Thapanun, Sawangjai, Phattarapong, Thuwajit, Peti, Boonsakan, Paisarn, Sripodok, Supasan, Veerakanjana, Kanyakorn, Luenam, Phoomraphee, Charngkaew, Komgrid, Pongpaibul, Ananya, Angkathunyakul, Napat, Hnoohom, Narit, Yuenyong, Sumeth, Thuwajit, Chanitra, Wilaiprasitporn, Theerawit
Patch classification models based on deep learning have been utilized in whole-slide images (WSI) of H&E-stained tissue samples to assist pathologists in grading follicular lymphoma patients. However, these approaches still require pathologists to manually identify centroblast cells and provide refined labels for optimal performance. To address this, we propose PseudoCell, an object detection framework to automate centroblast detection in WSI (source code is available at https://github.com/IoBT-VISTEC/PseudoCell.git). This framework incorporates centroblast labels from pathologists and combines them with pseudo-negative labels obtained from undersampled false-positive predictions using the cell's morphological features. By employing PseudoCell, pathologists' workload can be reduced as it accurately narrows down the areas requiring their attention during examining tissue. Depending on the confidence threshold, PseudoCell can eliminate 58.18-99.35% of non-centroblasts tissue areas on WSI. This study presents a practical centroblast prescreening method that does not require pathologists' refined labels for improvement. Detailed guidance on the practical implementation of PseudoCell is provided in the discussion section.
The Role of Subgroup Separability in Group-Fair Medical Image Classification
Jones, Charles, Roschewitz, Mรฉlanie, Glocker, Ben
We investigate performance disparities in deep classifiers. We find that the ability of classifiers to separate individuals into subgroups varies substantially across medical imaging modalities and protected characteristics; crucially, we show that this property is predictive of algorithmic bias. Through theoretical analysis and extensive empirical evaluation, we find a relationship between subgroup separability, subgroup disparities, and performance degradation when models are trained on data with systematic bias such as underdiagnosis. Our findings shed new light on the question of how models become biased, providing important insights for the development of fair medical imaging AI.
Density-based Feasibility Learning with Normalizing Flows for Introspective Robotic Assembly
Feng, Jianxiang, Atad, Matan, Rodrรญguez, Ismael, Durner, Maximilian, Gรผnnemann, Stephan, Triebel, Rudolph
Machine Learning (ML) models in Robotic Assembly Sequence Planning (RASP) need to be introspective on the predicted solutions, i.e. whether they are feasible or not, to circumvent potential efficiency degradation. Previous works need both feasible and infeasible examples during training. However, the infeasible ones are hard to collect sufficiently when re-training is required for swift adaptation to new product variants. In this work, we propose a density-based feasibility learning method that requires only feasible examples. Concretely, we formulate the feasibility learning problem as Out-of-Distribution (OOD) detection with Normalizing Flows (NF), which are powerful generative models for estimating complex probability distributions. Empirically, the proposed method is demonstrated on robotic assembly use cases and outperforms other single-class baselines in detecting infeasible assemblies. We further investigate the internal working mechanism of our method and show that a large memory saving can be obtained based on an advanced variant of NF.
Evaluating the Evaluators: Are Current Few-Shot Learning Benchmarks Fit for Purpose?
Shimabucoro, Luรญsa, Hospedales, Timothy, Gouk, Henry
The Few Shot-Learning (FSL) paradigm, which focuses on enabling models to generalise well with little data Numerous benchmarks for Few-Shot Learning through the use of transferred prior knowledge, has gained have been proposed in the last decade. However relevance in an attempt to overcome these challenges. A all of these benchmarks focus on performance significant amount of attention has been given to FSL and averaged over many tasks, and the question of related meta-learning research in the last decade (Wang how to reliably evaluate and tune models trained et al., 2020; Hospedales et al., 2021), with a large number of for individual tasks in this regime has not been methods and benchmarks proposed in application domains addressed. This paper presents the first investigation ranging from visual recognition systems for robots to identifying into task-level evaluation--a fundamental therapeutic properties of molecules (Xie et al., 2018; step when deploying a model. We measure the accuracy Stanley et al., 2021). of performance estimators in the few-shot setting, consider strategies for model selection, Even though many learning algorithms have been developed and examine the reasons for the failure of evaluators in this area and great efforts have been directed towards usually thought of as being robust. We improving model performance in FSL scenarios, the best conclude that cross-validation with a low number practices for how to evaluate models and design benchmarks of folds is the best choice for directly estimating for this paradigm remain relatively unexplored. In typical the performance of a model, whereas using bootstrapping academic benchmark setups, performance estimation often or cross validation with a large number relies on the existence of test ("query") sets that are several of folds is better for model selection purposes.
Through the Fairness Lens: Experimental Analysis and Evaluation of Entity Matching
Shahbazi, Nima, Danevski, Nikola, Nargesian, Fatemeh, Asudeh, Abolfazl, Srivastava, Divesh
Entity matching (EM) is a challenging problem studied by different communities for over half a century. Algorithmic fairness has also become a timely topic to address machine bias and its societal impacts. Despite extensive research on these two topics, little attention has been paid to the fairness of entity matching. Towards addressing this gap, we perform an extensive experimental evaluation of a variety of EM techniques in this paper. We generated two social datasets from publicly available datasets for the purpose of auditing EM through the lens of fairness. Our findings underscore potential unfairness under two common conditions in real-world societies: (i) when some demographic groups are overrepresented, and (ii) when names are more similar in some groups compared to others. Among our many findings, it is noteworthy to mention that while various fairness definitions are valuable for different settings, due to EM's class imbalance nature, measures such as positive predictive value parity and true positive rate parity are, in general, more capable of revealing EM unfairness.
GIT: Detecting Uncertainty, Out-Of-Distribution and Adversarial Samples using Gradients and Invariance Transformations
Lust, Julia, Condurache, Alexandru P.
Deep neural networks tend to make overconfident predictions and often require additional detectors for misclassifications, particularly for safety-critical applications. Existing detection methods usually only focus on adversarial attacks or out-of-distribution samples as reasons for false predictions. However, generalization errors occur due to diverse reasons often related to poorly learning relevant invariances. We therefore propose GIT, a holistic approach for the detection of generalization errors that combines the usage of gradient information and invariance transformations. The invariance transformations are designed to shift misclassified samples back into the generalization area of the neural network, while the gradient information measures the contradiction between the initial prediction and the corresponding inherent computations of the neural network using the transformed sample. Our experiments demonstrate the superior performance of GIT compared to the state-of-the-art on a variety of network architectures, problem setups and perturbation types.
Mainline Automatic Train Horn and Brake Performance Metric
This paper argues for the introduction of a mainline rail-oriented performance metric for driver-replacing on-board perception systems. Perception at the head of a train is divided into several subfunctions. This article presents a preliminary submetric for the obstacle detection subfunction. To the best of the author's knowledge, no other such proposal for obstacle detection exists. A set of submetrics for the subfunctions should facilitate the comparison of perception systems among each other and guide the measurement of human driver performance. It should also be useful for a standardized prediction of the number of accidents for a given perception system in a given operational design domain. In particular, for the proposal of the obstacle detection submetric, the professional readership is invited to provide their feedback and quantitative information to the author. The analysis results of the feedback will be published separately later.
How accurate are existing land cover maps for agriculture in Sub-Saharan Africa?
Kerner, Hannah, Nakalembe, Catherine, Yang, Adam, Zvonkov, Ivan, McWeeny, Ryan, Tseng, Gabriel, Becker-Reshef, Inbal
Satellite Earth observations (EO) can provide affordable and timely information for assessing crop conditions and food production. Such monitoring systems are essential in Africa, where there is high food insecurity and sparse agricultural statistics. EO-based monitoring systems require accurate cropland maps to provide information about croplands, but there is a lack of data to determine which of the many available land cover maps most accurately identify cropland in African countries. This study provides a quantitative evaluation and intercomparison of 11 publicly available land cover maps to assess their suitability for cropland classification and EO-based agriculture monitoring in Africa using statistically rigorous reference datasets from 8 countries. We hope the results of this study will help users determine the most suitable map for their needs and encourage future work to focus on resolving inconsistencies between maps and improving accuracy in low-accuracy regions.
Graph Contrastive Topic Model
Luo, Zheheng, Liu, Lei, Xie, Qianqian, Ananiadou, Sophia
Existing NTMs with contrastive learning suffer from the sample bias problem owing to the word frequency-based sampling strategy, which may result in false negative samples with similar semantics to the prototypes. In this paper, we aim to explore the efficient sampling strategy and contrastive learning in NTMs to address the aforementioned issue. We propose a new sampling assumption that negative samples should contain words that are semantically irrelevant to the prototype. Based on it, we propose the graph contrastive topic model (GCTM), which conducts graph contrastive learning (GCL) using informative positive and negative samples that are generated by the graph-based sampling strategy leveraging in-depth correlation and irrelevance among documents and words. In GCTM, we first model the input document as the document word bipartite graph (DWBG), and construct positive and negative word co-occurrence graphs (WCGs), encoded by graph neural networks, to express in-depth semantic correlation and irrelevance among words. Based on the DWBG and WCGs, we design the document-word information propagation (DWIP) process to perform the edge perturbation of DWBG, based on multi-hop correlations/irrelevance among documents and words. This yields the desired negative and positive samples, which will be utilized for GCL together with the prototypes to improve learning document topic representations and latent topics. We further show that GCL can be interpreted as the structured variational graph auto-encoder which maximizes the mutual information of latent topic representations of different perspectives on DWBG. Experiments on several benchmark datasets demonstrate the effectiveness of our method for topic coherence and document representation learning compared with existing SOTA methods.