Performance Analysis
Lightweight 3D Convolutional Neural Network for Schizophrenia diagnosis using MRI Images and Ensemble Bagging Classifier
Patro, P Supriya, Goel, Tripti, VaraPrasad, S A, Tanveer, M, Murugan, R
Structural alterations have been thoroughly investigated in the brain during the early onset of schizophrenia (SCZ) with the development of neuroimaging methods. The objective of the paper is an efficient classification of SCZ in 2 different classes: Cognitive Normal (CN), and SCZ using magnetic resonance imaging (MRI) images. This paper proposed a lightweight 3D convolutional neural network (CNN) based framework for SCZ diagnosis using MRI images. In the proposed model, lightweight 3D CNN is used to extract both spatial and spectral features simultaneously from 3D volume MRI scans, and classification is done using an ensemble bagging classifier. Ensemble bagging classifier contributes to preventing overfitting, reduces variance, and improves the model's accuracy. The proposed algorithm is tested on datasets taken from three benchmark databases available as open-source: MCICShare, COBRE, and fBRINPhase-II. These datasets have undergone preprocessing steps to register all the MRI images to the standard template and reduce the artifacts. The model achieves the highest accuracy 92.22%, sensitivity 94.44%, specificity 90%, precision 90.43%, recall 94.44%, F1-score 92.39% and G-mean 92.19% as compared to the current state-of-the-art techniques. The performance metrics evidenced the use of this model to assist the clinicians for automatic accurate diagnosis of SCZ.
Learning the shape of protein micro-environments with a holographic convolutional neural network
Pun, Michael N., Ivanov, Andrew, Bellamy, Quinn, Montague, Zachary, LaMont, Colin, Bradley, Philip, Otwinowski, Jakub, Nourmohammad, Armita
Proteins play a central role in biology from immune recognition to brain activity. While major advances in machine learning have improved our ability to predict protein structure from sequence, determining protein function from structure remains a major challenge. Here, we introduce Holographic Convolutional Neural Network (H-CNN) for proteins, which is a physically motivated machine learning approach to model amino acid preferences in protein structures. H-CNN reflects physical interactions in a protein structure and recapitulates the functional information stored in evolutionary data. H-CNN accurately predicts the impact of mutations on protein function, including stability and binding of protein complexes. Our interpretable computational model for protein structure-function maps could guide design of novel proteins with desired function.
Design and Validation of a Portable Machine Learning-Based Electronic Nose
Volatile organic compounds (VOCs) are chemicals emitted by various groups, such as foods, bacteria, and plants. While there are specific pathways and biological features significantly related to such VOCs, detection of these is achieved mostly by human odor testing or high-end methods such as gas chromatographyโmass spectrometry that can analyze the gaseous component. However, odor characterization can be quite helpful in the rapid classification of some samples in sufficient concentrations. Lower-cost metal-oxide gas sensors have the potential to allow the same type of detection with less training required. Here, we report a portable, battery-powered electronic nose system that utilizes multiple metal-oxide gas sensors and machine learning algorithms to detect and classify VOCs. An in-house circuit was designed with ten metal-oxide sensors and voltage dividers; an STM32 microcontroller was used for data acquisition with 12-bit analog-to-digital conversion. For classification of target samples, a supervised machine learning algorithm such as support vector machine (SVM) was applied to classify the VOCs based on the measurement results. The coefficient of variation (standard deviation divided by mean) of 8 of the 10 sensors stayed below 10%, indicating the excellent repeatability of these sensors. As a proof of concept, four different types of wine samples and three different oil samples were classified, and the training model reported 100% and 98% accuracy based on the confusion matrix analysis, respectively. When the trained model was challenged against new sets of data, sensitivity and specificity of 98.5% and 98.6% were achieved for the wine test and 96.3% and 93.3% for the oil test, respectively, when the SVM classifier was used. These results suggest that the metal-oxide sensors are suitable for usage in food authentication applications.
Logits are predictive of network type
We show that it is possible to predict which deep network has generated a given logit vector with accuracy well above chance. We utilize a number of networks on a dataset, initialized with random weights or pretrained weights, as well as fine-tuned networks. A classifier is then trained on the logit vectors of the trained set of this dataset to map the logit vector to the network index that has generated it. The classifier is then evaluated on the test set of the dataset. Results are better with randomly initialized networks, but also generalize to pretrained networks as well as fine-tuned ones. Classification accuracy is higher using unnormalized logits than normalized ones. We find that there is little transfer when applying a classifier to the same networks but with different sets of weights. In addition to help better understand deep networks and the way they encode uncertainty, we anticipate our finding to be useful in some applications (e.g. tailoring an adversarial attack for a certain type of network). Code is available at https://github.com/aliborji/logits.
Forecasting User Interests Through Topic Tag Predictions in Online Health Communities
Adishesha, Amogh Subbakrishna, Jakielaszek, Lily, Azhar, Fariha, Zhang, Peixuan, Honavar, Vasant, Ma, Fenglong, Belani, Chandra, Mitra, Prasenjit, Huang, Sharon Xiaolei
The increasing reliance on online communities for healthcare information by patients and caregivers has led to the increase in the spread of misinformation, or subjective, anecdotal and inaccurate or non-specific recommendations, which, if acted on, could cause serious harm to the patients. Hence, there is an urgent need to connect users with accurate and tailored health information in a timely manner to prevent such harm. This paper proposes an innovative approach to suggesting reliable information to participants in online communities as they move through different stages in their disease or treatment. We hypothesize that patients with similar histories of disease progression or course of treatment would have similar information needs at comparable stages. Specifically, we pose the problem of predicting topic tags or keywords that describe the future information needs of users based on their profiles, traces of their online interactions within the community (past posts, replies) and the profiles and traces of online interactions of other users with similar profiles and similar traces of past interaction with the target users. The result is a variant of the collaborative information filtering or recommendation system tailored to the needs of users of online health communities. We report results of our experiments on an expert curated data set which demonstrate the superiority of the proposed approach over the state of the art baselines with respect to accurate and timely prediction of topic tags (and hence information sources of interest).
Impact Learning: A Learning Method from Features Impact and Competition
Prottasha, Nusrat Jahan, Murad, Saydul Akbar, Muzahid, Abu Jafar Md, Rana, Masud, Kowsher, Md, Adhikary, Apurba, Biswas, Sujit, Bairagi, Anupam Kumar
Machine learning is the study of computer algorithms that can automatically improve based on data and experience. Machine learning algorithms build a model from sample data, called training data, to make predictions or judgments without being explicitly programmed to do so. A variety of wellknown machine learning algorithms have been developed for use in the field of computer science to analyze data. This paper introduced a new machine learning algorithm called impact learning. Impact learning is a supervised learning algorithm that can be consolidated in both classification and regression problems. It can furthermore manifest its superiority in analyzing competitive data. This algorithm is remarkable for learning from the competitive situation and the competition comes from the effects of autonomous features. It is prepared by the impacts of the highlights from the intrinsic rate of natural increase (RNI). We, moreover, manifest the prevalence of the impact learning over the conventional machine learning algorithm.
Confusion Matrix: How To Use It & Interpret Results [Examples]
A confusion matrix, as the name suggests, is a matrix of numbers that tell us where a model gets confused. It is a class-wise distribution of the predictive performance of a classification model--that is, the confusion matrix is an organized way of mapping the predictions to the original classes to which the data belong. This also implies that confusion matrices can only be used when the output distribution is known, i.e., in supervised learning frameworks. The confusion matrix not only allows the calculation of the accuracy of a classifier, be it the global or the class-wise accuracy, but also helps compute other important metrics that developers often use to evaluate their models. A confusion matrix computed for the same test set of a dataset, but using different classifiers, can also help compare their relative strengths and weaknesses and draw an inference about how they can be combined (ensemble learning) to obtain the optimal performance.
GRAIMATTER Green Paper: Recommendations for disclosure control of trained Machine Learning (ML) models from Trusted Research Environments (TREs)
Jefferson, Emily, Liley, James, Malone, Maeve, Reel, Smarti, Crespi-Boixader, Alba, Kerasidou, Xaroula, Tava, Francesco, McCarthy, Andrew, Preen, Richard, Blanco-Justicia, Alberto, Mansouri-Benssassi, Esma, Domingo-Ferrer, Josep, Beggs, Jillian, Chuter, Antony, Cole, Christian, Ritchie, Felix, Daly, Angela, Rogers, Simon, Smith, Jim
TREs are widely, and increasingly used to support statistical analysis of sensitive data across a range of sectors (e.g., health, police, tax and education) as they enable secure and transparent research whilst protecting data confidentiality. There is an increasing desire from academia and industry to train AI models in TREs. The field of AI is developing quickly with applications including spotting human errors, streamlining processes, task automation and decision support. These complex AI models require more information to describe and reproduce, increasing the possibility that sensitive personal data can be inferred from such descriptions. TREs do not have mature processes and controls against these risks. This is a complex topic, and it is unreasonable to expect all TREs to be aware of all risks or that TRE researchers have addressed these risks in AI-specific training. GRAIMATTER has developed a draft set of usable recommendations for TREs to guard against the additional risks when disclosing trained AI models from TREs. The development of these recommendations has been funded by the GRAIMATTER UKRI DARE UK sprint research project. This version of our recommendations was published at the end of the project in September 2022. During the course of the project, we have identified many areas for future investigations to expand and test these recommendations in practice. Therefore, we expect that this document will evolve over time.
Unintended Memorization and Timing Attacks in Named Entity Recognition Models
Ali, Rana Salal, Zhao, Benjamin Zi Hao, Asghar, Hassan Jameel, Nguyen, Tham, Wood, Ian David, Kaafar, Dali
Named entity recognition models (NER), are widely used for identifying named entities (e.g., individuals, locations, and other information) in text documents. Machine learning based NER models are increasingly being applied in privacy-sensitive applications that need automatic and scalable identification of sensitive information to redact text for data sharing. In this paper, we study the setting when NER models are available as a black-box service for identifying sensitive information in user documents and show that these models are vulnerable to membership inference on their training datasets. With updated pre-trained NER models from spaCy, we demonstrate two distinct membership attacks on these models. Our first attack capitalizes on unintended memorization in the NER's underlying neural network, a phenomenon NNs are known to be vulnerable to. Our second attack leverages a timing side-channel to target NER models that maintain vocabularies constructed from the training data. We show that different functional paths of words within the training dataset in contrast to words not previously seen have measurable differences in execution time. Revealing membership status of training samples has clear privacy implications, e.g., in text redaction, sensitive words or phrases to be found and removed, are at risk of being detected in the training dataset. Our experimental evaluation includes the redaction of both password and health data, presenting both security risks and privacy/regulatory issues. This is exacerbated by results that show memorization with only a single phrase. We achieved 70% AUC in our first attack on a text redaction use-case. We also show overwhelming success in the timing attack with 99.23% AUC. Finally we discuss potential mitigation approaches to realize the safe use of NER models in light of the privacy and security implications of membership inference attacks.
Uncertainty Quantification for Rule-Based Models
Rule-based classification models described in the language of logic directly predict boolean values, rather than modeling a probability and translating it into a prediction as done in statistical models. The vast majority of existing uncertainty quantification approaches rely on models providing continuous output not available to rule-based models. In this work, we propose an uncertainty quantification framework in the form of a meta-model that takes any binary classifier with binary output as a black box and estimates the prediction accuracy of that base model at a given input along with a level of confidence on that estimation. The confidence is based on how well that input region is explored and is designed to work in any OOD scenario. We demonstrate the usefulness of this uncertainty model by building an abstaining classifier powered by it and observing its performance in various scenarios.