Performance Analysis
Cross-institution text mining to uncover clinical associations: a case study relating social factors and code status in intensive care medicine
Sushil, Madhumita, Butte, Atul J., Schuit, Ewoud, van Smeden, Maarten, Leeuwenberg, Artuur M.
Objective: Text mining of clinical notes embedded in electronic medical records is increasingly used to extract patient characteristics otherwise not or only partly available, to assess their association with relevant health outcomes. As manual data labeling needed to develop text mining models is resource intensive, we investigated whether off-the-shelf text mining models developed at external institutions, together with limited within-institution labeled data, could be used to reliably extract study variables to conduct association studies. Materials and Methods: We developed multiple text mining models on different combinations of within-institution and external-institution data to extract social factors from discharge reports of intensive care patients. Subsequently, we assessed the associations between social factors and having a do-not-resuscitate/intubate code. Results: Important differences were found between associations based on manually labeled data compared to text-mined social factors in three out of five cases. Adopting external-institution text mining models using manually labeled within-institution data resulted in models with higher F1-scores, but not in meaningfully different associations. Discussion: While text mining facilitated scaling analyses to larger samples leading to discovering a larger number of associations, the estimates may be unreliable. Confirmation is needed with better text mining models, ideally on a larger manually labeled dataset. Conclusion: The currently used text mining models were not sufficiently accurate to be used reliably in an association study. Model adaptation using within-institution data did not improve the estimates. Further research is needed to set conditions for reliable use of text mining in medical research.
Sensor data-driven analysis for identification of causal relationships between exposure to air pollution and respiratory rate in asthmatics
According to the Lancet report on the global burden of disease published in October 2020, air pollution is among the five highest risk factors for global health, reducing life expectancy on average by 20 months. This paper describes a data-driven method for establishing causal relationships within and between two multivariate time series data streams derived from wearable sensors: personal exposure to airborne particulate matter of aerodynamic sizes less than 2.5um (PM2.5) gathered from the Airspeck monitor worn on the person and continuous respiratory rate (breaths per minute) measured by the Respeck monitor worn as a plaster on the chest. Results are presented for a cohort of 113 asthmatic adolescents using the PCMCI+ algorithm to learn the short-term causal relationships between lags of \pm exposure and respiratory rate. We consider causal effects up to a maximum delay of 8 hours, using data at both a 1 minute and 15 minute resolution in different experiments. For the first time a personalised exposure-response relationship between PM2.5 exposure and respiratory rate has been demonstrated to exist for short-term effects in asthmatic adolescents during their everyday lives. Our results lead to recommendations for work on specific open problems in causal discovery, to increase the feasibility of this approach for similar epidemiology studies in the future.
Distinguish Sense from Nonsense: Out-of-Scope Detection for Virtual Assistants
Qian, Cheng, Qi, Haode, Wang, Gengyu, Kunc, Ladislav, Potdar, Saloni
Out of Scope (OOS) detection in Conversational AI solutions enables a chatbot to handle a conversation gracefully when it is unable to make sense of the end-user query. Accurately tagging a query as out-of-domain is particularly hard in scenarios when the chatbot is not equipped to handle a topic which has semantic overlap with an existing topic it is trained on. We propose a simple yet effective OOS detection method that outperforms standard OOS detection methods in a real-world deployment of virtual assistants. We discuss the various design and deployment considerations for a cloud platform solution to train virtual assistants and deploy them at scale. Additionally, we propose a collection of datasets that replicates real-world scenarios and show comprehensive results in various settings using both offline and online evaluation metrics.
Error-related Potential Variability: Exploring the Effects on Classification and Transferability
Brain-Computer Interfaces (BCI) have allowed for direct communication from the brain to external applications for the automatic detection of cognitive processes such as error recognition. Error-related potentials (ErrPs) are a particular brain signal elicited when one commits or observes an erroneous event. However, due to the noisy properties of the brain and recording devices, ErrPs vary from instance to instance as they are combined with an assortment of other brain signals, biological noise, and external noise, making the classification of ErrPs a non-trivial problem. Recent works have revealed particular cognitive processes such as awareness, embodiment, and predictability that contribute to ErrP variations. In this paper, we explore the performance of classifier transferability when trained on different ErrP variation datasets generated by varying the levels of awareness and embodiment for a given task. In particular, we look at transference between observational and interactive ErrP categories when elicited by similar and differing tasks. Our empirical results provide an exploratory analysis into the ErrP transferability problem from a data perspective.
A New Deep Boosted CNN and Ensemble Learning based IoT Malware Detection
Khan, Saddam Hussain, Ullah, Wasi
Security issues are threatened in various types of networks, especially in the Internet of Things (IoT) environment that requires early detection. IoT is the network of real-time devices like home automation systems and can be controlled by open-source android devices, which can be an open ground for attackers. Attackers can access the network credentials, initiate a different kind of security breach, and compromises network control. Therefore, timely detecting the increasing number of sophisticated malware attacks is the challenge to ensure the credibility of network protection. In this regard, we have developed a new malware detection framework, Deep Squeezed-Boosted and Ensemble Learning (DSBEL), comprised of novel Squeezed-Boosted Boundary-Region Split-Transform-Merge (SB-BR-STM) CNN and ensemble learning. The proposed STM block employs multi-path dilated convolutional, Boundary, and regional operations to capture the homogenous and heterogeneous global malicious patterns. Moreover, diverse feature maps are achieved using transfer learning and multi-path-based squeezing and boosting at initial and final levels to learn minute pattern variations. Finally, the boosted discriminative features are extracted from the developed deep SB-BR-STM CNN and provided to the ensemble classifiers (SVM, MLP, and AdabooSTM1) to improve the hybrid learning generalization. The performance analysis of the proposed DSBEL framework and SB-BR-STM CNN against the existing techniques have been evaluated by the IOT_Malware dataset on standard performance measures. Evaluation results show progressive performance as 98.50% accuracy, 97.12% F1-Score, 91.91% MCC, 95.97 % Recall, and 98.42 % Precision. The proposed malware analysis framework is robust and helpful for the timely detection of malicious activity and suggests future strategies
Calibrated Data-Dependent Constraints with Exact Satisfaction Guarantees
Xue, Songkai, Sun, Yuekai, Yurochkin, Mikhail
We consider the task of training machine learning models with data-dependent constraints. Such constraints often arise as empirical versions of expected value constraints that enforce fairness or stability goals. We reformulate data-dependent constraints so that they are calibrated: enforcing the reformulated constraints guarantees that their expected value counterparts are satisfied with a user-prescribed probability. The resulting optimization problem is amendable to standard stochastic optimization algorithms, and we demonstrate the efficacy of our method on a fairness-sensitive classification task where we wish to guarantee the classifier's fairness (at test time).
Interpretable and Scalable Graphical Models for Complex Spatio-temporal Processes
This thesis focuses on data that has complex spatio-temporal structure and on probabilistic graphical models that learn the structure in an interpretable and scalable manner. We target two research areas of interest: Gaussian graphical models for tensor-variate data and summarization of complex time-varying texts using topic models. This work advances the state-of-the-art in several directions. First, it introduces a new class of tensor-variate Gaussian graphical models via the Sylvester tensor equation. Second, it develops an optimization technique based on a fast-converging proximal alternating linearized minimization method, which scales tensor-variate Gaussian graphical model estimations to modern big-data settings. Third, it connects Kronecker-structured (inverse) covariance models with spatio-temporal partial differential equations (PDEs) and introduces a new framework for ensemble Kalman filtering that is capable of tracking chaotic physical systems. Fourth, it proposes a modular and interpretable framework for unsupervised and weakly-supervised probabilistic topic modeling of time-varying data that combines generative statistical models with computational geometric methods. Throughout, practical applications of the methodology are considered using real datasets. This includes brain-connectivity analysis using EEG data, space weather forecasting using solar imaging data, longitudinal analysis of public opinions using Twitter data, and mining of mental health related issues using TalkLife data. We show in each case that the graphical modeling framework introduced here leads to improved interpretability, accuracy, and scalability.
What artificial intelligence might teach us about the origin of human language
This study explores an interesting pattern emerging from research that combines artificial intelligence with sound symbolism. In these studies, supervised machine learning algorithms are trained to classify samples based on the sounds of referent names. Machine learning algorithms are efficient learners of sound symbolism, but they tend to bias one category over the other. The pattern is this: when a category arguably represents greater threat, the algorithms tend to overpredict to that category. A hypothesis, framed by error management theory, is presented that proposes that this may be evidence of an adaptation to preference cautious behaviour. This hypothesis is tested by constructing extreme gradient boosted (XGBoost) models using the sounds that make up the names of Chinese, Japanese and Korean Pokemon and observing classification error distribution.
A Review on the effectiveness of Dimensional Reduction with Computational Forensics: An Application on Malware Analysis
Da Naing, Aye Thaw, Guan, Justin Soh Beng, Win, Yarzar Shwe, Pan, Jonathan
The Android operating system is pervasively adopted as the operating system platform of choice for smart devices. However, the strong adoption has also resulted in exponential growth in the number of Android based malicious software or malware. To deal with such cyber threats as part of cyber investigation and digital forensics, computational techniques in the form of machine learning algorithms are applied for such malware identification, detection and forensics analysis. However, such Computational Forensics modelling techniques are constrained the volume, velocity, variety and veracity of the malware landscape. This in turn would affect its identification and detection effectiveness. Such consequence would inherently induce the question of sustainability with such solution approach. One approach to optimise effectiveness is to apply dimensional reduction techniques like Principal Component Analysis with the intent to enhance algorithmic performance. In this paper, we evaluate the effectiveness of the application of Principle Component Analysis on Computational Forensics task of detecting Android based malware. We applied our research hypothesis to three different datasets with different machine learning algorithms. Our research result showed that the dimensionally reduced dataset would result in a measure of degradation in accuracy performance.