Performance Analysis
Using Machine Learning to Develop Smart Reflex Testing Protocols
McDermott, Matthew, Dighe, Anand, Szolovits, Peter, Luo, Yuan, Baron, Jason
Objective: Reflex testing protocols allow clinical laboratories to perform second line diagnostic tests on existing specimens based on the results of initially ordered tests. Reflex testing can support optimal clinical laboratory test ordering and diagnosis. In current clinical practice, reflex testing typically relies on simple "if-then" rules; however, this limits their scope since most test ordering decisions involve more complexity than a simple rule will allow. Here, using the analyte ferritin as an example, we propose an alternative machine learning-based approach to "smart" reflex testing with a wider scope and greater impact than traditional rule-based approaches. Methods: Using patient data, we developed a machine learning model to predict whether a patient getting CBC testing will also have ferritin testing ordered, consider applications of this model to "smart" reflex testing, and evaluate the model by comparing its performance to possible rule-based approaches. Results: Our underlying machine learning models performed moderately well in predicting ferritin test ordering and demonstrated greater suitability to reflex testing than rule-based approaches. Using chart review, we demonstrate that our model may improve ferritin test ordering. Finally, as a secondary goal, we demonstrate that ferritin test results are missing not at random (MNAR), a finding with implications for unbiased imputation of missing test results. Conclusions: Machine learning may provide a foundation for new types of reflex testing with enhanced benefits for clinical diagnosis and laboratory utilization management.
Learning to be Fair: A Consequentialist Approach to Equitable Decision-Making
Chohlas-Wood, Alex, Coots, Madison, Zhu, Henry, Brunskill, Emma, Goel, Sharad
In the dominant paradigm for designing equitable machine learning systems, one works to ensure that model predictions satisfy various fairness criteria, such as parity in error rates across race, gender, and other legally protected traits. That approach, however, typically ignores the downstream decisions and outcomes that predictions affect, and, as a result, can induce unexpected harms. Here we present an alternative framework for fairness that directly anticipates the consequences of decisions. Stakeholders first specify preferences over the possible outcomes of an algorithmically informed decision-making process. For example, lenders may prefer extending credit to those most likely to repay a loan, while also preferring similar lending rates across neighborhoods. One then searches the space of decision policies to maximize the specified utility. We develop and describe a method for efficiently learning these optimal policies from data for a large family of expressive utility functions, facilitating a more holistic approach to equitable decision-making.
Epic-Sounds: A Large-scale Dataset of Actions That Sound
Huh, Jaesung, Chalk, Jacob, Kazakos, Evangelos, Damen, Dima, Zisserman, Andrew
We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos. We propose an annotation pipeline where annotators temporally label distinguishable audio segments and describe the action that could have caused this sound. We identify actions that can be discriminated purely from audio, through grouping these free-form descriptions of audio into classes. For actions that involve objects colliding, we collect human annotations of the materials of these objects (e.g. a glass object being placed on a wooden surface), which we verify from visual labels, discarding ambiguities. Overall, EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes as well as 39.2k non-categorised segments. We train and evaluate two state-of-the-art audio recognition models on our dataset, highlighting the importance of audio-only labels and the limitations of current models to recognise actions that sound.
Developing Hands-on Labs for Source Code Vulnerability Detection with AI
As the role of information and communication technologies gradually increases in our lives, source code security becomes a significant issue to protect against malicious attempts. Furthermore, with the advent of data-driven techniques, there is now a growing interest in leveraging machine learning and natural language processing (NLP) as a source code assurance method to build trustworthy systems. Therefore, training our future software developers to write secure source code is in high demand. In this thesis, we propose a framework including learning modules and handson labs to guide future IT professionals towards developing secure programming habits and mitigating source code vulnerabilities at the early stages of the software development lifecycle. In this thesis, our goal is to design learning modules with a set of hands-on labs that will introduce students to secure programming practices using source code and log file analysis tools to predict/identify vulnerabilities. In a Secure Coding Education framework called (SeCodEd) we will (1) improve students' skills and awareness on source code vulnerabilities, detection tools, and mitigation techniques; (2) integrate concepts of source code vulnerabilities from Function, API, and library level to bad programming habits and practices; (3) leverage deep learning, NLP and static analysis tools for log file analysis to introduce the root cause of source code vulnerabilities.
Using novel data and ensemble models to improve automated labeling of Sustainable Development Goals
Wulff, Dirk U., Meier, Dominik S., Mata, Rui
A number of labeling systems based on text have been proposed to help monitor work on the United Nations (UN) Sustainable Development Goals (SDGs). Here, we present a systematic comparison of systems using a variety of text sources and show that systems differ considerably in their specificity (i.e., true-positive rate) and sensitivity (i.e., true-negative rate), have systematic biases (e.g., are more sensitive to specific SDGs relative to others), and are susceptible to the type and amount of text analyzed. We then show that an ensemble model that pools labeling systems alleviates some of these limitations, exceeding the labeling performance of all currently available systems. We conclude that researchers and policymakers should care about the choice of labeling system and that ensemble methods should be favored when drawing conclusions about the absolute and relative prevalence of work on the SDGs based on automated methods.
Convolutional Neural Network for Breast Cancer Classification
Click here to read the full story with my Friend Link! Breast cancer is the second most common cancer in women and men worldwide. In 2012, it represented about 12 percent of all new cancer cases and 25 percent of all cancers in women. Breast cancer starts when cells in the breast begin to grow out of control. These cells usually form a tumor that can often be seen on an x-ray or felt as a lump. The tumor is malignant (cancer) if the cells can grow into (invade) surrounding tissues or spread (metastasize) to distant areas of the body.
CT Study Says Deep Learning Model Could Help Differentiate Between Acute Diverticulitis and Colon Carcinoma
Noting that overlapping imaging features on contrast-enhanced computed tomography (CT) can make it challenging to differentiate between acute diverticulitis and colon cancer, researchers say an emerging deep learning model may provide enhanced sensitivity and specificity for these conditions. In a retrospective study recently published in JAMA Network Open, researchers developed and tested a three-dimensional (3D) convolutional neural network (CNN) for 585 patients (mean age of 63.2) who underwent surgery for colon cancer or acute diverticulitis between July 1, 2005 and October 1, 2020, had venous phase CT imaging within 60 days prior to surgery and had segmental wall thickening in the colon that was independent of disease stage. In comparison to mean sensitivity and specificity rates of 77.6 percent and 81.6 percent, respectively, for radiologist readers, the study authors noted an 83.3 percent sensitivity rate and an 86.6 percent specificity rate for the 3D CNN model. The combination of the deep learning model and radiologist assessment resulted in an eight percent increase in sensitivity (85.6 percent) and a 9.7 percent increase in specificity (91.3 percent) over radiologist assessments, according to the study findings. The study authors also noted the reduction of false-negative rates with the 3D CNN model.
[2301.12670] A deep-learning search for technosignatures of 820 nearby stars
The goal of the Search for Extraterrestrial Intelligence (SETI) is to quantify the prevalence of technological life beyond Earth via their "technosignatures". One theorized technosignature is narrowband Doppler drifting radio signals. The principal challenge in conducting SETI in the radio domain is developing a generalized technique to reject human radio frequency interference (RFI). Here, we present the most comprehensive deep-learning based technosignature search to date, returning 8 promising ETI signals of interest for re-observation as part of the Breakthrough Listen initiative. The search comprises 820 unique targets observed with the Robert C. Byrd Green Bank Telescope, totaling over 480, hr of on-sky data. We implement a novel beta-Convolutional Variational Autoencoder to identify technosignature candidates in a semi-unsupervised manner while keeping the false positive rate manageably low. This new approach presents itself as a leading solution in accelerating SETI and other transient research into the age of data-driven astronomy.
A Prescriptive Learning Analytics Framework: Beyond Predictive Modelling and onto Explainable AI with Prescriptive Analytics and ChatGPT
A significant body of recent research in the field of Learning Analytics has focused on leveraging machine learning approaches for predicting at-risk students in order to initiate timely interventions and thereby elevate retention and completion rates. The overarching feature of the majority of these research studies has been on the science of prediction only. The component of predictive analytics concerned with interpreting the internals of the models and explaining their predictions for individual cases to stakeholders has largely been neglected. Additionally, works that attempt to employ data-driven prescriptive analytics to automatically generate evidence-based remedial advice for at-risk learners are in their infancy. eXplainable AI is a field that has recently emerged providing cutting-edge tools which support transparent predictive analytics and techniques for generating tailored advice for at-risk students. This study proposes a novel framework that unifies both transparent machine learning as well as techniques for enabling prescriptive analytics, while integrating the latest advances in large language models. This work practically demonstrates the proposed framework using predictive models for identifying at-risk learners of programme non-completion. The study then further demonstrates how predictive modelling can be augmented with prescriptive analytics on two case studies in order to generate human-readable prescriptive feedback for those who are at risk using ChatGPT.
Bayesian Bilinear Neural Network for Predicting the Mid-price Dynamics in Limit-Order Book Markets
Magris, Martin, Shabani, Mostafa, Iosifidis, Alexandros
The prediction of financial markets is a challenging yet important task. In modern electronically-driven markets, traditional time-series econometric methods often appear incapable of capturing the true complexity of the multi-level interactions driving the price dynamics. While recent research has established the effectiveness of traditional machine learning (ML) models in financial applications, their intrinsic inability to deal with uncertainties, which is a great concern in econometrics research and real business applications, constitutes a major drawback. Bayesian methods naturally appear as a suitable remedy conveying the predictive ability of ML methods with the probabilistically-oriented practice of econometric research. By adopting a state-of-the-art second-order optimization algorithm, we train a Bayesian bilinear neural network with temporal attention, suitable for the challenging time-series task of predicting mid-price movements in ultra-high-frequency limit-order book markets. We thoroughly compare our Bayesian model with traditional ML alternatives by addressing the use of predictive distributions to analyze errors and uncertainties associated with the estimated parameters and model forecasts. Our results underline the feasibility of the Bayesian deep-learning approach and its predictive and decisional advantages in complex econometric tasks, prompting future research in this direction.