Collaborating Authors

performance metric

AUA 2020: Automated Performance Metrics to Predict Continence Recovery After Robotic Radical Prostatectomy Utilizing Machine Learning


The robot-assisted radical prostatectomy was segmented into 12 steps, and for each step, 41 validated automated performance metrics were reported. The predictive models were trained with three data sets: 1) 492 automated performance metrics; 2) 16 clinicopathological data (for example prostate volume, Gleason score); 3) automated performance metrics plus clinicopathological data. The authors utilized a random forest model (800 trees) to predict continence recovery (no pads or one safety pad) at three and six months after surgery. The prediction accuracy was estimated through a 10-fold cross-validation process. The area under the curve (AUC) and standard error (SE) was used to estimate prediction accuracy. Finally, the out-of-bag Gini index was used to rank the variables of importance.

CSMOUTE: Combined Synthetic Oversampling and Undersampling Technique for Imbalanced Data Classification Machine Learning

In this paper we propose two novel data-level algorithms for handling data imbalance in the classification task: first of all a Synthetic Minority Undersampling Technique (SMUTE), which leverages the concept of interpolation of nearby instances, previously introduced in the oversampling setting in SMOTE, and secondly a Combined Synthetic Oversampling and Undersampling Technique (CSMOUTE), which integrates SMOTE oversampling with SMUTE undersampling. The results of the conducted experimental study demonstrate the usefulness of both the SMUTE and the CSMOUTE algorithms, especially when combined with a more complex classifiers, namely MLP and SVM, and when applied on a datasets consisting of a large number of outliers. This leads us to a conclusion that the proposed approach shows promise for further extensions accommodating local data characteristics, a direction discussed in more detail in the paper.

AI4COVID-19: AI Enabled Preliminary Diagnosis for COVID-19 from Cough Samples via an App Machine Learning

Inability to test at scale has become Achille's heel in humanity's ongoing war against COVID-19 pandemic. An agile, scalable and cost-effective testing, deployable at a global scale, can act as a game changer in this war. To address this challenge, building on the promising results of our prior work on cough-based diagnosis of a motley of respiratory diseases, we develop an Artificial Intelligence (AI)-based test for COVID-19 preliminary diagnosis. The test is deployable at scale through a mobile app named AI4COVID-19. The AI4COVID-19 app requires 2-second cough recordings of the subject. By analyzing the cough samples through an AI engine running in the cloud, the app returns a preliminary diagnosis within a minute. Unfortunately, cough is common symptom of over two dozen non-COVID-19 related medical conditions. This makes the COVID-19 diagnosis from cough alone an extremely challenging problem. We solve this problem by developing a novel multi-pronged mediator centered risk-averse AI architecture that minimizes misdiagnosis. At the time of writing, our AI engine can distinguish between COVID-19 patient coughs and several types of non-COVID-19 coughs with over 90% accuracy. AI4COVID-19's performance is likely to improve as more and better data becomes available. This paper presents a proof of concept to encourage controlled clinical trials and serves as a call for labeled cough data. AI4COVID-19 is not designed to compete with clinical testing. Instead, it offers a complementing tele-testing tool deployable anytime, anywhere, by anyone, so clinical-testing and treatment can be channeled to those who need it the most, thereby saving more lives.

Probabilistic Diagnostic Tests for Degradation Problems in Supervised Learning Artificial Intelligence

Several studies point out different causes of performance degradation in supervised machine learning. Problems such as class imbalance, overlapping, small-disjuncts, noisy labels, and sparseness limit accuracy in classification algorithms. Even though a number of approaches either in the form of a methodology or an algorithm try to minimize performance degradation, they have been isolated efforts with limited scope. Most of these approaches focus on remediation of one among many problems, with experimental results coming from few datasets and classification algorithms, insufficient measures of prediction power, and lack of statistical validation for testing the real benefit of the proposed approach. This paper consists of two main parts: In the first part, a novel probabilistic diagnostic model based on identifying signs and symptoms of each problem is presented. Thereby, early and correct diagnosis of these problems is to be achieved in order to select not only the most convenient remediation treatment but also unbiased performance metrics. Secondly, the behavior and performance of several supervised algorithms are studied when training sets have such problems. Therefore, prediction of success for treatments can be estimated across classifiers.

Automated detection of pitting and stress corrosion cracks in used nuclear fuel dry storage canisters using residual neural networks Machine Learning

Nondestructive evaluation methods play an important role in ensuring component integrity and safety in many industries. Operator fatigue can play a critical role in the reliability of such methods. This is important for inspecting high value assets or assets with a high consequence of failure, such as aerospace and nuclear components. Recent advances in convolution neural networks can support and automate these inspection efforts. This paper proposes using residual neural networks (ResNets) for real-time detection of pitting and stress corrosion cracking, with a focus on dry storage canisters housing used nuclear fuel. The proposed approach crops nuclear canister images into smaller tiles, trains a ResNet on these tiles, and classifies images as corroded or intact using the per-image count of tiles predicted as corroded by the ResNet. The results demonstrate that such a deep learning approach allows to detect the locus of corrosion cracks via smaller tiles, and at the same time to infer with high accuracy whether an image comes from a corroded canister. Thereby, the proposed approach holds promise to automate and speed up nuclear fuel canister inspections, to minimize inspection costs, and to partially replace human-conducted onsite inspections, thus reducing radiation doses to personnel.

Metrics and methods for robustness evaluation of neural networks with generative models Machine Learning

Recent studies have shown that modern deep neural network classifiers are easy to fool, assuming that an adversary is able to slightly modify their inputs. Many papers have proposed adversarial attacks, defenses and methods to measure robustness to such adversarial perturbations. However, most commonly considered adversarial examples are based on l p -bounded perturbations in the input space of the neural network, which are unlikely to arise naturally. Recently, especially in computer vision, researchers discovered "natural" or "semantic" perturbations, such as rotations, changes of brightness, or more high-level changes, but these perturbations have not yet been systematically utilized to measure the performance of classifiers. In this paper, we propose several metrics to measure robustness of classifiers to natural adversarial examples, and methods to evaluate them. These metrics, called latent space performance metrics, are based on the ability of generative models to capture probability distributions, and are defined in their latent spaces. On three image classification case studies, we evaluate the proposed metrics for several classifiers, including ones trained in conventional and robust ways. We find that the latent counterparts of adversarial robustness are associated with the accuracy of the classifier rather than its conventional adversarial robustness, but the latter is still reflected on the properties of found latent perturbations. In addition, our novel method of finding latent adversarial perturbations demonstrates that these perturbations are often perceptually small.

Consistent Binary Classification with Generalized Performance Metrics

Neural Information Processing Systems

Performance metrics for binary classification are designed to capture tradeoffs between four fundamental population quantities: true positives, false positives, true negatives and false negatives. Despite significant interest from theoretical and applied communities, little is known about either optimal classifiers or consistent algorithms for optimizing binary classification performance metrics beyond a few special cases. We consider a fairly large family of performance metrics given by ratios of linear combinations of the four fundamental population quantities. This family includes many well known binary classification metrics such as classification accuracy, AM measure, F-measure and the Jaccard similarity coefficient as special cases. Our analysis identifies the optimal classifiers as the sign of the thresholded conditional probability of the positive class, with a performance metric-dependent threshold.

Google Cloud AutoML Vision for Medical Image Classification


The concepts of neural architecture search and transfer learning are used under the hood to find the best network architecture and the optimal hyperparameter configuration that minimizes the loss function of the model. This article uses Google Cloud AutoML Vision to develop an end-to-end medical image classification model for Pneumonia Detection using Chest X-Ray Images. The dataset is hosted on Kaggle and can be accessed at Chest X-Ray Images (Pneumonia). Go to the cloud console: Setup Project APIs, permissions and Cloud Storage bucket to store the image files for modeling and other assets.

A Generalized Flow for B2B Sales Predictive Modeling: An Azure Machine Learning Approach Machine Learning

-- Predicting s ales opportunities outcome is a core to successful business management and revenue forecasting . Conventionally, this prediction has relied mostly on subjective human evaluations in the process of business to business (B2B) sales decision making. Here, we proposed a practical Machine Learning (ML) workflow to empower B2B sales outcome (win/lose) pre diction within a cloud - based computing platform: Microsoft Azure Machine Learning Service (Azure ML). This workflow consists of two pipelines: 1) a n ML pipeline that trains probabilistic predictive models in parallel on the closed sales opportunities data enhanced with an extensive feature engineering procedure for automated selection and parameterization of an optimal ML model and 2) a Prediction pipeline that uses the optimal ML model to estimate the likelihood of win n ing new sales opportunities as well a s predicting their outcome using optimized decision boundaries. The p erformance of the proposed workflow was evaluated on a real sales dataset of a B2B consulting firm. In the Business to Business (B2B) commerce, companies compete to win high - valued sales opportunities to maximize their profitability. In this regard, a key factor for maintain ing a successful B2B business is the task of determining the outcome of sales opportunities.

DYNOTEARS: Structure Learning from Time-Series Data Machine Learning

In this paper, we revisit the structure learning problem for dynamic Bayesian networks and propose a method that simultaneously estimates contemporaneous (intra-slice) and time-lagged (inter-slice) relationships between variables in a time-series. Our approach is score-based, and revolves around minimizing a penalized loss subject to an acyclicity constraint. To solve this problem, we leverage a recent algebraic result characterizing the acyclicity constraint as a smooth equality constraint. The resulting algorithm, which we call DYNOTEARS, outperforms other methods on simulated data, especially in high-dimensions as the number of variables increases. We also apply this algorithm on real datasets from two different domains, finance and molecular biology, and analyze the resulting output. Compared to state-of-the-art methods for learning dynamic Bayesian networks, our method is both scalable and accurate on real data. The simple formulation, and competitive performance of our method make it suitable for a variety of problems where one seeks to learn connections between variables across time.