Goto

Collaborating Authors

 Performance Analysis


Saliency Grafting: Innocuous Attribution-Guided Mixup with Calibrated Label Mixing

arXiv.org Artificial Intelligence

The Mixup scheme suggests mixing a pair of samples to create an augmented training sample and has gained considerable attention recently for improving the generalizability of neural networks. A straightforward and widely used extension of Mixup is to combine with regional dropout-like methods: removing random patches from a sample and replacing it with the features from another sample. Albeit their simplicity and effectiveness, these methods are prone to create harmful samples due to their randomness. To address this issue, 'maximum saliency' strategies were recently proposed: they select only the most informative features to prevent such a phenomenon. However, they now suffer from lack of sample diversification as they always deterministically select regions with maximum saliency, injecting bias into the augmented data. In this paper, we present, a novel, yet simple Mixup-variant that captures the best of both worlds. Our idea is two-fold. By stochastically sampling the features and 'grafting' them onto another sample, our method effectively generates diverse yet meaningful samples. Its second ingredient is to produce the label of the grafted sample by mixing the labels in a saliency-calibrated fashion, which rectifies supervision misguidance introduced by the random sampling procedure. Our experiments under CIFAR, Tiny-ImageNet, and ImageNet datasets show that our scheme outperforms the current state-of-the-art augmentation strategies not only in terms of classification accuracy, but is also superior in coping under stress conditions such as data corruption and object occlusion.


Kalendar AI wants its sales bots to win your next customers – TechCrunch

#artificialintelligence

Kalendar AI, a San Francisco-based startup that's been building on top of GPT-3's language model -- developing a SaaS for automating lead generation and sales outreach to make it easier for companies to get initial meetings with prospective customers -- has raised $3.2 million in pre-seed funding from 500 Startups; The Lean Startup author, Eric Ries; VC firms Village Global and Metaplanet; and 20 angel investors (including CEOs of "popular" but undisclosed companies). "Our AI technology writes personalized invitations to ideal customers with personalized decks -- inviting them to take a meeting," explains founder and CEO Ravi Vadrevu. The SaaS was launched in February this year, although the startup itself -- which is called Kriya Inc -- was founded back in 2017 and had been bootstrapping prior to raising this pre-seed. The idea for the b2b product is to automate the time-consuming and expensive process of sales outreach, including locating and pitching leads, as well as to offer tools to streamline and enhance initial sales meetings. Kalendar AI claims to have amassed a database of 340M "ideal customer profiles" upon which it unleashes its AI sales rep bots to send "personalized" pitches (including "interactive presentations that convert into one-click meetings") to likely looking customers. "Our solution brings down the time to initiate a conversation to book an appointment from 7 days to 30 seconds from a sales perspective," claims Vadrevu, who also argues there are big productivity wins from a marketing perspective vs other channels.


Confusion Matrix

#artificialintelligence

A Confusion matrix is an N x N matrix used for evaluating the performance of a classification model, where N is the number of target classes. The matrix compares the actual target values with those predicted by the machine learning model. This gives us a view of how well our classification model is performing and what kinds of errors it is making. A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class.


Do You See What I See? Capabilities and Limits of Automated Multimedia Content Analysis

arXiv.org Artificial Intelligence

The ever-increasing amount of user-generated content online has led, in recent years, to an expansion in research and investment in automated content analysis tools. Scrutiny of automated content analysis has accelerated during the COVID-19 pandemic, as social networking services have placed a greater reliance on these tools due to concerns about health risks to their moderation staff from in-person work. At the same time, there are important policy debates around the world about how to improve content moderation while protecting free expression and privacy. In order to advance these debates, we need to understand the potential role of automated content analysis tools. This paper explains the capabilities and limitations of tools for analyzing online multimedia content and highlights the potential risks of using these tools at scale without accounting for their limitations. It focuses on two main categories of tools: matching models and computer prediction models. Matching models include cryptographic and perceptual hashing, which compare user-generated content with existing and known content. Predictive models (including computer vision and computer audition) are machine learning techniques that aim to identify characteristics of new or previously unknown content.


Selecting the suitable resampling strategy for imbalanced data classification regarding dataset properties

arXiv.org Artificial Intelligence

In many application domains such as medicine, information retrieval, cybersecurity, social media, etc., datasets used for inducing classification models often have an unequal distribution of the instances of each class. This situation, known as imbalanced data classification, causes low predictive performance for the minority class examples. Thus, the prediction model is unreliable although the overall model accuracy can be acceptable. Oversampling and undersampling techniques are well-known strategies to deal with this problem by balancing the number of examples of each class. However, their effectiveness depends on several factors mainly related to data intrinsic characteristics, such as imbalance ratio, dataset size and dimensionality, overlapping between classes or borderline examples. In this work, the impact of these factors is analyzed through a comprehensive comparative study involving 40 datasets from different application areas. The objective is to obtain models for automatic selection of the best resampling strategy for any dataset based on its characteristics. These models allow us to check several factors simultaneously considering a wide range of values since they are induced from very varied datasets that cover a broad spectrum of conditions. This differs from most studies that focus on the individual analysis of the characteristics or cover a small range of values. In addition, the study encompasses both basic and advanced resampling strategies that are evaluated by means of eight different performance metrics, including new measures specifically designed for imbalanced data classification. The general nature of the proposal allows the choice of the most appropriate method regardless of the domain, avoiding the search for special purpose techniques that could be valid for the target data.


TrialGraph: Machine Intelligence Enabled Insight from Graph Modelling of Clinical Trials

arXiv.org Artificial Intelligence

A major impediment to successful drug development is the complexity, cost, and scale of clinical trials. The detailed internal structure of clinical trial data can make conventional optimization difficult to achieve. Recent advances in machine learning, specifically graph-structured data analysis, have the potential to enable significant progress in improving the clinical trial design. TrialGraph seeks to apply these methodologies to produce a proof-of-concept framework for developing models which can aid drug development and benefit patients. In this work, we first introduce a curated clinical trial data set compiled from the CT.gov, AACT and TrialTrove databases (n=1191 trials; representing one million patients) and describe the conversion of this data to graph-structured formats. We then detail the mathematical basis and implementation of a selection of graph machine learning algorithms, which typically use standard machine classifiers on graph data embedded in a low-dimensional feature space. We trained these models to predict side effect information for a clinical trial given information on the disease, existing medical conditions, and treatment. The MetaPath2Vec algorithm performed exceptionally well, with standard Logistic Regression, Decision Tree, Random Forest, Support Vector, and Neural Network classifiers exhibiting typical ROC-AUC scores of 0.85, 0.68, 0.86, 0.80, and 0.77, respectively. Remarkably, the best performing classifiers could only produce typical ROC-AUC scores of 0.70 when trained on equivalent array-structured data. Our work demonstrates that graph modelling can significantly improve prediction accuracy on appropriate datasets. Successive versions of the project that refine modelling assumptions and incorporate more data types can produce excellent predictors with real-world applications in drug development.


Multiple testing -- how should you adjust?

#artificialintelligence

Multiple testing adjustment has gained in popularity with large scale datasets used for exploratory purposes. It is now a key consideration in statistical inference problems.


Identification of Twitter Bots Based on an Explainable Machine Learning Framework: The US 2020 Elections Case Study

arXiv.org Artificial Intelligence

Twitter is one of the most popular social networks attracting millions of users, while a considerable proportion of online discourse is captured. It provides a simple usage framework with short messages and an efficient application programming interface (API) enabling the research community to study and analyze several aspects of this social network. However, the Twitter usage simplicity can lead to malicious handling by various bots. The malicious handling phenomenon expands in online discourse, especially during the electoral periods, where except the legitimate bots used for dissemination and communication purposes, the goal is to manipulate the public opinion and the electorate towards a certain direction, specific ideology, or political party. This paper focuses on the design of a novel system for identifying Twitter bots based on labeled Twitter data. To this end, a supervised machine learning (ML) framework is adopted using an Extreme Gradient Boosting (XGBoost) algorithm, where the hyper-parameters are tuned via cross-validation. Our study also deploys Shapley Additive Explanations (SHAP) for explaining the ML model predictions by calculating feature importance, using the game theoretic-based Shapley values. Experimental evaluation on distinct Twitter datasets demonstrate the superiority of our approach, in terms of bot detection accuracy, when compared against a recent state-of-the-art Twitter bot detection method.


A real-time spatiotemporal AI model analyzes skill in open surgical videos

arXiv.org Artificial Intelligence

Surgery offers the potential to treat and cure many diseases, but complications from surgical procedures remain the third highest cause of death globally [1]. Recent studies have shown that surgeons rated as higher-skilled via peer grading have lower rates of complications and death [2, 3]. Systems to evaluate surgical skill and provide feedback to improve technique could have a dramatic effect on the variation that exists in the field. Unfortunately, current approaches for evaluating surgical procedures and technique are primarily qualitative and do not have the ability to scale or even identify the elements of surgeon judgment that drive patient outcomes [4]. Artificial intelligence (AI) in the form of computer vision algorithms could provide scalable, automated analysis of surgical behaviors from video streams. AI could serve as an additional coach for surgical trainees and as an expert colleague for experienced surgeons [5]. However, the development of computer vision for open surgery--the dominant form of surgery defined as traditional, non-camera-based surgical techniques [6]--has been limited by two factors: the complexity of the AI task, and a lack of diverse and sizable training datasets [7]. Our work shows that a multi-task, spatiotemporal AI model, trained on multi-institutional data from numerous surgeons, has the potential to provide consistent analysis and feedback without the bias of any particular surgeon's experience.


Artificial Intelligence Ethics and Safety: practical tools for creating "good" models

arXiv.org Artificial Intelligence

The AI Robotics Ethics Society (AIRES) is a non-profit organization founded in 2018 by Aaron Hui to promote awareness and the importance of ethical implementation and regulation of AI. AIRES is now an organization with chapters at universities such as UCLA (Los Angeles), USC (University of Southern California), Caltech (California Institute of Technology), Stanford University, Cornell University, Brown University, and the Pontifical Catholic University of Rio Grande do Sul (Brazil). AIRES at PUCRS is the first international chapter of AIRES, and as such, we are committed to promoting and enhancing the AIRES Mission. Our mission is to focus on educating the AI leaders of tomorrow in ethical principles to ensure that AI is created ethically and responsibly. As there are still few proposals for how we should implement ethical principles and normative guidelines in the practice of AI system development, the goal of this work is to try to bridge this gap between discourse and praxis. Between abstract principles and technical implementation. In this work, we seek to introduce the reader to the topic of AI Ethics and Safety. At the same time, we present several tools to help developers of intelligent systems develop "good" models. This work is a developing guide published in English and Portuguese. Contributions and suggestions are welcome.