Performance Analysis
False Positive Reduction by Actively Mining Negative Samples for Pulmonary Nodule Detection in Chest Radiographs
Park, Sejin, Hwang, Woochan, Jung, Kyu Hwan, Seo, Joon Beom, Kim, Namkug
Generating large quantities of quality labeled data in medical imaging is very time consuming and expensive. The performance of supervised algorithms for various tasks on imaging has improved drastically over the years, however the availability of data to train these algorithms have become one of the main bottlenecks for implementation. To address this, we propose a semi-supervised learning method where pseudo-negative labels from unlabeled data are used to further refine the performance of a pulmonary nodule detection network in chest radiographs. After training with the proposed network, the false positive rate was reduced to 0.1266 from 0.4864 while maintaining sensitivity at 0.89.
Neural State Classification for Hybrid Systems
Phan, Dung, Paoletti, Nicola, Zhang, Timothy, Grosu, Radu, Smolka, Scott A., Stoller, Scott D.
We introduce the State Classification Problem (SCP) for hybrid systems, and present Neural State Classification (NSC) as an efficient solution technique. SCP generalizes the model checking problem as it entails classifying each state $s$ of a hybrid automaton as either positive or negative, depending on whether or not $s$ satisfies a given time-bounded reachability specification. This is an interesting problem in its own right, which NSC solves using machine-learning techniques, Deep Neural Networks in particular. State classifiers produced by NSC tend to be very efficient (run in constant time and space), but may be subject to classification errors. To quantify and mitigate such errors, our approach comprises: i) techniques for certifying, with statistical guarantees, that an NSC classifier meets given accuracy levels; ii) tuning techniques, including a novel technique based on adversarial sampling, that can virtually eliminate false negatives (positive states classified as negative), thereby making the classifier more conservative. We have applied NSC to six nonlinear hybrid system benchmarks, achieving an accuracy of 99.25% to 99.98%, and a false-negative rate of 0.0033 to 0, which we further reduced to 0.0015 to 0 after tuning the classifier. We believe that this level of accuracy is acceptable in many practical applications, and that these results demonstrate the promise of the NSC approach.
Robustness to fundamental uncertainty in AGI alignment
The AGI alignment problem has a bimodal distribution of outcomes with most outcomes clustering around the poles of total success and existential, catastrophic failure. Consequently, attempts to solve AGI alignment should, all else equal, prefer false negatives (ignoring research programs that would have been successful) to false positives (pursuing research programs that will unexpectedly fail). Thus, we propose adopting a policy of responding to points of metaphysical and practical uncertainty associated with the alignment problem by limiting and choosing necessary assumptions to reduce the risk false positives. Herein we explore in detail some of the relevant points of uncertainty that AGI alignment research hinges on and consider how to reduce false positives in response to them.
A Probabilistic Theory of Supervised Similarity Learning for Pointwise ROC Curve Optimization
Vogel, Robin, Bellet, Aurรฉlien, Clรฉmenรงon, Stรฉphan
The performance of many machine learning techniques depends on the choice of an appropriate similarity or distance measure on the input space. Similarity learning (or metric learning) aims at building such a measure from training data so that observations with the same (resp. different) label are as close (resp. far) as possible. In this paper, similarity learning is investigated from the perspective of pairwise bipartite ranking, where the goal is to rank the elements of a database by decreasing order of the probability that they share the same label with some query data point, based on the similarity scores. A natural performance criterion in this setting is pointwise ROC optimization: maximize the true positive rate under a fixed false positive rate. We study this novel perspective on similarity learning through a rigorous probabilistic framework. The empirical version of the problem gives rise to a constrained optimization formulation involving U-statistics, for which we derive universal learning rates as well as faster rates under a noise assumption on the data distribution. We also address the large-scale setting by analyzing the effect of sampling-based approximations. Our theoretical results are supported by illustrative numerical experiments.
Look, what's that over there? Sophos nips Windows DNS DLL false positive in the bud
A Windows operating system library was wrongly identified as malware by Sophos's antivirus scanner for some users on Tuesday. The main gripe seemed to be bogus alerts generated by the software, rather than crashed systems, a not infrequent side-effect of erroneously putting Windows library files into quarantine. Influential UK infosec geezer Kevin Beaumont highlighted the cockup, and soon after El Reg began prodding Sophos about the issue, the false positives were cancelled and normality was restored. How much pain, confusion and general inconvenience did the incident cause? Probably not much, it would seem, mostly because the issue was quickly resolved.
Continuous Authentication of Smartphones Based on Application Usage
Mahbub, Upal, Komulainen, Jukka, Ferreira, Denzil, Chellappa, Rama
An empirical investigation of active/continuous authentication for smartphones is presented in this paper by exploiting users' unique application usage data, i.e., distinct patterns of use, modeled by a Markovian process. Variations of Hidden Markov Models (HMMs) are evaluated for continuous user verification, and challenges due to the sparsity of session-wise data, an explosion of states, and handling unforeseen events in the test data are tackled. Unlike traditional approaches, the proposed formulation does not depend on the top N-apps, rather uses the complete app-usage information to achieve low latency. Through experimentation, empirical assessment of the impact of unforeseen events, i.e., unknown applications and unforeseen observations, on user verification is done via a modified edit-distance algorithm for simple sequence matching. It is found that for enhanced verification performance, unforeseen events should be incorporated in the models by adopting smoothing techniques with HMMs. For validation, extensive experiments on two distinct datasets are performed. The marginal smoothing technique is the most effective for user verification in terms of equal error rate (EER) and with a sampling rate of 1/30s^{-1} and 30 minutes of historical data, and the method is capable of detecting an intrusion within ~2.5 minutes of application use.
Confidence Intervals for Testing Disparate Impact in Fair Learning
Besse, Philippe, del Barrio, Eustasio, Gordaliza, Paula, Loubes, Jean-Michel
We provide the asymptotic distribution of the major indexes used in the statistical literature to quantify disparate treatment in machine learning. We aim at promoting the use of confidence intervals when testing the so-called group disparate impact. We illustrate on some examples the importance of using confidence intervals and not a single value.
InclusiveFaceNet: Improving Face Attribute Detection with Race and Gender Diversity
Ryu, Hee Jung, Adam, Hartwig, Mitchell, Margaret
We demonstrate an approach to face attribute detection that retains or improves attribute detection accuracy across gender and race subgroups by learning demographic information prior to learning the attribute detection task. The system, which we call InclusiveFaceNet, detects face attributes by transferring race and gender representations learned from a held-out dataset of public race and gender identities. Leveraging learned demographic representations while withholding demographic inference from the downstream face attribute detection task preserves potential users' demographic privacy while resulting in some of the best reported numbers to date on attribute detection in the Faces of the World and CelebA datasets.
Debunking Google's Death AI - Predictive Analytics Times - machine learning & data science news
Editor's note: Although this author absolves the researchers (from Google) and blames only the journalists for the widespread false claims of a 95% accuracy level for mortality prediction, note that the research paper itself does indeed use the word "accuracy" multiple times as a synonym of AUROC, thus "starting it" among the non-technical or less technical journalists at large. Having my newsfeed cluttered with articles about Google creating an AI that beats hospitals by predicting death with 95% accuracy (or some other erroneous claim), I dug up the original research paper to fact check this wondrous new advancement. Many of said articles used this quote from the abstract (academia's equivalent of a paperback blurb): These models outperformed traditional, clinically used predictive models in all cases. We believe that this approach can be used to create accurate and scaleable predictions for a variety of clinical scenarios. To the best of our knowledge, our models outperform existing EHR (Electronic Health Record) models in the medical literature.
Data Infrastructure and Approaches for Ontology-Based Drug Repurposing
Boyer, Stephen, Griffin, Thomas, Swaminathan, Sarath, Clarkson, Kenneth L., Zubarev, Dmitry
IBM Almaden Research Center, 650 Harry Road, San Jose, California 95136 Abstract We report development of a data infrastructure for drug repurposing that takes advantage of two currently available chemical ontologies. The data infrastructure includes a database of compoundtarget associations augmented with molecular ontological labels. It also contains two computational tools for prediction of new associations. We describe two drug-repurposing systems: one, Nascent Ontological Information Retrieval for Drug Repurposing (NOIR-DR), based on an information retrieval strategy, and another, based on nonnegative matrix factorization together with compound similarity, that was inspired by recommender systems. We report the performance of both tools on a drug-repurposing task. 1 Introduction Drug repurposing is an efficient strategy for drug discovery, where new targets or activities are found for known drugs [1-5]. Drug repurposing requires the efficient representation of existing information about the activity of chemical compounds as drugs, and the development of algorithms that leverage such information and propose new indications.