Performance Analysis
Risk Ranked Recall: Collision Safety Metric for Object Detection Systems in Autonomous Vehicles
Bansal, Ayoosh, Singh, Jayati, Verucchi, Micaela, Caccamo, Marco, Sha, Lui
Abstract--Commonly used metrics for evaluation of object detection systems (precision, recall, mAP) do not give complete information about their suitability of use in safety critical tasks, like obstacle detection for collision avoidance in Autonomous Vehicles (AV). Ranks are assigned based on an objective cyber-physical model for the risk of collision. Recall is measured for each rank. A front view scene from BDD100K [1] dataset with 4 labeled vehicles. Intuitively, the closer vehicles are more important to detect than those farther away.
Description and Discussion on DCASE 2021 Challenge Task 2: Unsupervised Anomalous Sound Detection for Machine Condition Monitoring under Domain Shifted Conditions
Kawaguchi, Yohei, Imoto, Keisuke, Koizumi, Yuma, Harada, Noboru, Niizumi, Daisuke, Dohi, Kota, Tanabe, Ryo, Purohit, Harsh, Endo, Takashi
We present the task description and discussion on the results of the DCASE 2021 Challenge Task 2. Last year, we organized unsupervised anomalous sound detection (ASD) task; identifying whether the given sound is normal or anomalous without anomalous training data. In this year, we organize an advanced unsupervised ASD task under domain-shift conditions which focuses on the inevitable problem for the practical use of ASD systems. The main challenge of this task is to detect unknown anomalous sounds where the acoustic characteristics of the training and testing samples are different, i.e. domain-shifted. This problem is frequently occurs due to changes in seasons, manufactured products, and/or environmental noise. After the challenge submission deadline, we will add challenge results and analysis of the submissions.
How Confusion Matrix is useful in solving Cyber Crimes
In Machine Learning we feed the features into our model and get the output in the form of probabilities. But how can we measure the accuracy and effectiveness of that model? This is where confusion matrix comes into the play. Confusion matrix is used to describe the performance of the classification model. First let us understand about the confusion matrix.
MemStream: Memory-Based Anomaly Detection in Multi-Aspect Streams with Concept Drift
Bhatia, Siddharth, Jain, Arjit, Srivastava, Shivin, Kawaguchi, Kenji, Hooi, Bryan
Given a stream of entries over time in a multi-aspect data setting where concept drift is present, how can we detect anomalous activities? Most of the existing unsupervised anomaly detection approaches seek to detect anomalous events in an offline fashion and require a large amount of data for training. This is not practical in real-life scenarios where we receive the data in a streaming manner and do not know the size of the stream beforehand. Thus, we need a data-efficient method that can detect and adapt to changing data trends, or concept drift, in an online manner. In this work, we propose MemStream, a streaming multi-aspect anomaly detection framework, allowing us to detect unusual events as they occur while being resilient to concept drift. We leverage the power of a denoising autoencoder to learn representations and a memory module to learn the dynamically changing trend in data without the need for labels. We prove the optimum memory size required for effective drift handling. Furthermore, MemStream makes use of two architecture design choices to be robust to memory poisoning. Experimental results show the effectiveness of our approach compared to state-of-the-art streaming baselines using 2 synthetic datasets and 11 real-world datasets.
Accurate and robust Shapley Values for explaining predictions and focusing on local important variables
Amoukou, Salim I., Brunel, Nicolas J-B., Salaรผn, Tangi
Although Shapley Values (SV) are widely used in explainable AI, they can be poorly understood and estimated, which implies that their analysis may lead to spurious inferences and explanations. As a starting point, we remind an invariance principle for SV and derive the correct approach for computing the SV of categorical variables that are particularly sensitive to the encoding used. In the case of tree-based models, we introduce two estimators of Shapley Values that exploit efficiently the tree structure and are more accurate than state-of-the-art methods. For interpreting additive explanations, we recommend to filter the non-influential variables and to compute the Shapley Values only for groups of influential variables. For this purpose, we use the concept of "Same Decision Probability" (SDP) that evaluates the robustness of a prediction when some variables are missing. This prior selection procedure produces sparse additive explanations easier to visualize and analyse. Simulations and comparisons are performed with state-of-the-art algorithm, and show the practical gain of our approach.
Bias Mitigation of Face Recognition Models Through Calibration
Salvador, Tiago, Cairns, Stephanie, Voleti, Vikram, Marshall, Noah, Oberman, Adam
Face recognition models suffer from bias: for example, the probability of a false positive (incorrect face match) strongly depends on sensitive attributes like ethnicity. As a result, these models may disproportionately and negatively impact minority groups when used in law enforcement. In this work, we introduce the Bias Mitigation Calibration (BMC) method, which (i) increases model accuracy (improving the state-of-the-art), (ii) produces fairly-calibrated probabilities, (iii) significantly reduces the gap in the false positive rates, and (iv) does not require knowledge of the sensitive attribute.
Ethical AI, Monetizing False Negatives and Growing Total Addressable Market
What if I told you that companies that don't embrace Ethical AI are leaving significant amounts of "Money on the Table"; that they are not only missing out on potentially profitable customers, but that over time they are eroding their Total Addressable Market (TAM)? Do I have your attention now? After I published the blog "The Ethical AI Application Pyramid", a question from Karrie Sullivan coupled with a mentoring session with the startup unfog.ai "If your AI model doesn't take into consideration the ultimate outcomes of the AI model's False Negatives, then confirmation bias in the AI model could set in and eventually the company's Total Addressable Market (TAM) could shrink to a point where the business might no longer be viable." Yea, not only is Ethical AI the right thing to do from a cultural and society perspective, but there are direct bottom-line financial ramifications if your AI models are not learning and adapting from the AI model's False Negatives.
Confusion Matrix In Cyber Security
In today's article I'm going to explain all about Intrusion detection system in cyber security, confusion matrix, how it is used in IDS, how it is impacting in cyber security with example .So let's get started to this amazing topic. In today's technological world where everything is going to digitalized everything is online now. Along with this the most important thing is data and data security. All activities we do on internet, what we searched,what we post, what we buy, which site we visited all this data is stored in datacenters servers. This all data must be secured from hackers and any kind of data loss.
Cybersecurity: When we talk about the confusion matrix
Confusion Matrix The Confusion Matrix is a table that summarizes the number of true and false predictions made by a classifier. It is used to measure the performance of a classification model. It can be used to assess the performance of a classification model by calculating performance indicators such as accuracy, precision, recall, and F1 score. If you are working with an unbalanced dataset, you had better use the confusion matrix as the endpoint for your machine learning model. Here are the basic terms that will help us identify the metrics we are looking for.
Virtual Screening of Pharmaceutical Compounds with hERG Inhibitory Activity (Cardiotoxicity) using Ensemble Learning
Sarkar, Aditya, Bhavsar, Arnav
In silico prediction of cardiotoxicity with high sensitivity and specificity for potential drug molecules can be of immense value. Hence, building machine learning classification models, based on some features extracted from the molecular structure of drugs, which are capable of efficiently predicting cardiotoxicity is critical. In this paper, we consider the application of various machine learning approaches, and then propose an ensemble classifier for the prediction of molecular activity on a Drug Discovery Hackathon (DDH) (1st reference) dataset. We have used only 2-D descriptors of SMILE notations for our prediction. Our ensemble classification uses 5 classifiers (2 Random Forest Classifiers, 2 Support Vector Machines and a Dense Neural Network) and uses Max-Voting technique and Weighted-Average technique for final decision. Introduction It is well known that drug discovery is complex, long drawn, and requires interdisciplinary expertise to discover new molecules. Drug safety is an important issue in the process of drug discovery. Failure in clinical trials in the 2000s was majorly due to efficacy and safety (approx 30%) (Kola, I. and Landis, J., 2004). One important aspect of drug safety is drug toxicity.