Performance Analysis
Drama at 'The View': COVID tests were 'false positives,' co-host reveals
The'Outnumbered' panel reacts to Sunny Hostin and Ana Navarro being pulled from the set moments before the vice president was set to arrive Ana Navarro, one of two co-hosts who were pulled from ABC's "The View" live on air Friday due to positive COVID-19 tests, has since revealed the results that caused the chaos were false positives. Producers informed Navarro and Sunny Hostin in their earpieces halfway through Friday's broadcast that they would have to leave the Hot Topics table, leaving Joy Behar and Sara Haines to conduct the rest of the show on their own. The remaining hosts often struggled to kill time, at one point taking questions from the audience, but often not being able to hear the questions that were muffled by their masks. Friday's drama was even more pronounced considering Navarro and Hostin were pulled just as Vice President Kamala Harris was on her way to the studio for an in-person interview. Even though Harris made it to the building, producers explained her appearance would end up taking place remotely from a separate room out of precaution.
Johns Hopkins has developed a lung cancer blood test
Powered by artificial intelligence, a new lung cancer blood test developed at Johns Hopkins, combined with other metrics, correctly identified 94% of cancer cases in almost 800 patients. The lung cancer blood test, published in Nature Communications, searches for tiny fragments of DNA released by the tumor cells. The AI looks for patterns in this shattered DNA, rather than looking for specific pieces of cancer DNA like other blood tests in development, New Atlas explained. Lung cancer kills the most people in the world, the authors note, "largely due to the late stage at diagnosis where treatments are less effective than at earlier stages" -- and lung cancer rates are increasing, worldwide. "We believe that a blood test, or'liquid biopsy,' for lung cancer could be a good way to enhance screening efforts, because it would be easy to do, broadly accessible, and cost-effective," study first author Dimitrios Mathios said. The DNA difference: Blood tests for cancer typically focus on finding pieces of mutated tumor DNA.
Distributionally Robust Multiclass Classification and Applications in Deep CNN Image Classifiers
Chen, Ruidi, Hao, Boran, Paschalidis, Ioannis
We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR), which could tolerate data contaminated by outliers. The DRO framework uses a probabilistic ambiguity set defined as a ball of distributions that are close to the empirical distribution of the training set in the sense of the Wasserstein metric. We relax the DRO formulation into a regularized learning problem whose regularizer is a norm of the coefficient matrix. We establish out-of-sample performance guarantees for the solutions to our model, offering insights on the role of the regularizer in controlling the prediction error. We apply the proposed method in rendering deep CNN-based image classifiers robust to random and adversarial attacks. Specifically, using the MNIST and CIFAR-10 datasets, we demonstrate reductions in test error rate by up to 78.8% and loss by up to 90.8%. We also show that with a limited number of perturbed images in the training set, our method can improve the error rate by up to 49.49% and the loss by up to 68.93% compared to Empirical Risk Minimization (ERM), converging faster to an ideal loss/error rate as the number of perturbed images increases.
Anomalous Edge Detection in Edge Exchangeable Social Network Models
Luo, Rui, Nettasinghe, Buddhika, Krishnamurthy, Vikram
This paper studies detecting anomalous edges in directed graphs that model social networks. We exploit edge exchangeability as a criterion for distinguishing anomalous edges from normal edges. Then we present an anomaly detector based on conformal prediction theory; this detector has a guaranteed upper bound for false positive rate. In numerical experiments, we show that the proposed algorithm achieves superior performance to baseline methods.
Intersectional Group Fairness in Machine Learning
At the ML Fairness Summit, we welcomed Fiddler Data Scientist, Lรฉa Genuit to discuss intersectional group fairness. As more companies adopt AI, more people question the impact AI creates on society, especially on algorithmic fairness. Instead, they hold a binary view of fairness, e.g., protected vs. unprotected groups. In the below blog, Lea covers the latest research in research on intersectional group fairness. Before explaining why, the first question should be how do you detect and mitigate bias in European models to avoid a bad experience?
Modelling the transition to a low-carbon energy supply
A transition to a low-carbon electricity supply is crucial to limit the impacts of climate change. Reducing carbon emissions could help prevent the world from reaching a tipping point, where runaway emissions are likely. Runaway emissions could lead to extremes in weather conditions around the world -- especially in problematic regions unable to cope with these conditions. However, the movement to a low-carbon energy supply can not happen instantaneously due to the existing fossil-fuel infrastructure and the requirement to maintain a reliable energy supply. Therefore, a low-carbon transition is required, however, the decisions various stakeholders should make over the coming decades to reduce these carbon emissions are not obvious. This is due to many long-term uncertainties, such as electricity, fuel and generation costs, human behaviour and the size of electricity demand. A well choreographed low-carbon transition is, therefore, required between all of the heterogenous actors in the system, as opposed to changing the behaviour of a single, centralised actor. The objective of this thesis is to create a novel, open-source agent-based model to better understand the manner in which the whole electricity market reacts to different factors using state-of-the-art machine learning and artificial intelligence methods. In contrast to other works, this thesis looks at both the long-term and short-term impact that different behaviours have on the electricity market by using these state-of-the-art methods.
Equality of opportunity in travel behavior prediction with deep neural networks and discrete choice models
Zheng, Yunhan, Wang, Shenhao, Zhao, Jinhua
Although researchers increasingly adopt machine learning to model travel behavior, they predominantly focus on prediction accuracy, ignoring the ethical challenges embedded in machine learning algorithms. This study introduces an important missing dimension - computational fairness - to travel behavior analysis. We first operationalize computational fairness by equality of opportunity, then differentiate between the bias inherent in data and the bias introduced by modeling. We then demonstrate the prediction disparities in travel behavior modeling using the 2017 National Household Travel Survey (NHTS) and the 2018-2019 My Daily Travel Survey in Chicago. Empirically, deep neural network (DNN) and discrete choice models (DCM) reveal consistent prediction disparities across multiple social groups: both over-predict the false negative rate of frequent driving for the ethnic minorities, the low-income and the disabled populations, and falsely predict a higher travel burden of the socially disadvantaged groups and the rural populations than reality. Comparing DNN with DCM, we find that DNN can outperform DCM in prediction disparities because of DNN's smaller misspecification error. To mitigate prediction disparities, this study introduces an absolute correlation regularization method, which is evaluated with synthetic and real-world data. The results demonstrate the prevalence of prediction disparities in travel behavior modeling, and the disparities still persist regarding a variety of model specifics such as the number of DNN layers, batch size and weight initialization. Since these prediction disparities can exacerbate social inequity if prediction results without fairness adjustment are used for transportation policy making, we advocate for careful consideration of the fairness problem in travel behavior modeling, and the use of bias mitigation algorithms for fair transport decisions.
Finetuning Transformer Models to Build ASAG System
Research towards creating systems for automatic grading of student answers to quiz and exam questions in educational settings has been ongoing since 1966. Over the years, the problem was divided into many categories. Among them, grading text answers were divided into short answer grading, and essay grading. The goal of this work was to develop an ML-based short answer grading system. I hence built a system which uses finetuning on Roberta Large Model pretrained on STS benchmark dataset and have also created an interface to show the production readiness of the system. I evaluated the performance of the system on the Mohler extended dataset and SciEntsBank Dataset. The developed system achieved a Pearsons Correlation of 0.82 and RMSE of 0.7 on the Mohler Dataset which beats the SOTA performance on this dataset which is correlation of 0.805 and RMSE of 0.793. Additionally, Pearsons Correlation of 0.79 and RMSE of 0.56 was achieved on the SciEntsBank Dataset, which only reconfirms the robustness of the system. A few observations during achieving these results included usage of batch size of 1 produced better results than using batch size of 16 or 32 and using huber loss as loss function performed well on this regression task. The system was tried and tested on train and validation splits using various random seeds and still has been tweaked to achieve a minimum of 0.76 of correlation and a maximum 0.15 (out of 1) RMSE on any dataset.
Sample Efficient Model Evaluation
Yilmaz, Emine, Hayes, Peter, Habib, Raza, Burgess, Jordan, Barber, David
Labelling data is a major practical bottleneck in training and testing classifiers. Given a collection of unlabelled data points, we address how to select which subset to label to best estimate test metrics such as accuracy, $F_1$ score or micro/macro $F_1$. We consider two sampling based approaches, namely the well-known Importance Sampling and we introduce a novel application of Poisson Sampling. For both approaches we derive the minimal error sampling distributions and how to approximate and use them to form estimators and confidence intervals. We show that Poisson Sampling outperforms Importance Sampling both theoretically and experimentally.
Optimization-based Causal Estimation from Heterogenous Environments
Yin, Mingzhang, Wang, Yixin, Blei, David M.
This paper presents a new optimization approach to causal estimation. Given data that contains covariates and an outcome, which covariates are causes of the outcome, and what is the strength of the causality? In classical machine learning (ML), the goal of optimization is to maximize predictive accuracy. However, some covariates might exhibit a non-causal association to the outcome. Such spurious associations provide predictive power for classical ML, but they prevent us from causally interpreting the result. This paper proposes CoCo, an optimization algorithm that bridges the gap between pure prediction and causal inference. CoCo leverages the recently-proposed idea of environments, datasets of covariates/response where the causal relationships remain invariant but where the distribution of the covariates changes from environment to environment. Given datasets from multiple environments -- and ones that exhibit sufficient heterogeneity -- CoCo maximizes an objective for which the only solution is the causal solution. We describe the theoretical foundations of this approach and demonstrate its effectiveness on simulated and real datasets. Compared to classical ML and existing methods, CoCo provides more accurate estimates of the causal model.