Interpretability and effectiveness are two essential and indispensable requirements for adopting machine learning methods in reality. In this paper, we propose a knowledge distillation based decision trees extension, dubbed rectified decision trees (ReDT), to explore the possibility of fulfilling those requirements simultaneously. Specifically, we extend the splitting criteria and the ending condition of the standard decision trees, which allows training with soft labels while preserving the deterministic splitting paths. We then train the ReDT based on the soft label distilled from a well-trained teacher model through a novel jackknife-based method. Accordingly, ReDT preserves the excellent interpretable nature of the decision trees while having a relatively good performance. The effectiveness of adopting soft labels instead of hard ones is also analyzed empirically and theoretically. Surprisingly, experiments indicate that the introduction of soft labels also reduces the model size compared with the standard decision trees from the aspect of the total nodes and rules, which is an unexpected gift from the `dark knowledge' distilled from the teacher model.

Incipient anomalies present milder symptoms compared to severe ones, and are more difficult to detect and diagnose due to their close resemblance to normal operating conditions. The lack of incipient anomaly examples in the training data can pose severe risks to anomaly detection methods that are built upon Machine Learning (ML) techniques, because these anomalies can be easily mistaken as normal operating conditions. To address this challenge, we propose to utilize the uncertainty information available from ensemble learning to identify potential misclassified incipient anomalies. We show in this paper that ensemble learning methods can give improved performance on incipient anomalies and identify common pitfalls in these models through extensive experiments on two real-world datasets. Then, we discuss how to design more effective ensemble models for detecting incipient anomalies.

Incipient anomalies present milder symptoms compared to severe ones, and are more difficult to detect and diagnose due to their close resemblance to normal operating conditions. The lack of incipient anomaly examples in the training data can pose severe risks to anomaly detection methods that are built upon Machine Learning (ML) techniques, because these anomalies can be easily mistaken as normal operating conditions. To address this challenge, we propose to utilize the uncertainty information available from ensemble learning to identify potential misclassified incipient anomalies. We show in this paper that ensemble learning methods can give improved performance on incipient anomalies and identify common pitfalls in these models through extensive experiments on two real-world datasets. Then, we discuss how to design more effective ensemble models for detecting incipient anomalies.

Inferring the structure of a causal model with feedback loops from observational data is a notoriously difficult--if not impossible--problem, particularly if one also seeks to guard against presence of latent confounders [9, 29]. We consider this problem for linear causal models given by mixed graphs (or path diagrams) with directed and bidirected edges. As detailed in Section 2, the vertices of such a graph correspond to the observed variables, and the directed edges encode structural equations that relate these variables up to stochastic noise. The bidirected edges indicate possible correlations among the noise terms, as may be induced by latent confounders. Much work has gone into algorithms that exploit conditional independence relations for learning the structure of causal models, or rather suitable equivalence classes of graphs encoding this structure; see, e.g., [10, 17, 18, 24, 25] or also the review of Spirtes and Zhang in [21, §18]. While methods have been developed that use information about conditional independence relations also in settings with feedback loops or latent variables, there is an inherent limitation to this approach as causal models with feedback loops or latent variables can generally not be characterized using conditional independence constraints alone [8, 27, 31, 32]. Alternatively, structure learning can be approached using score-based search techniques; see, e.g., [3, 28, 30].

Machine learning promises to revolutionize clinical decision making and diagnosis. In medical diagnosis a doctor aims to explain a patient’s symptoms by determining the diseases causing them. However, existing machine learning approaches to diagnosis are purely associative, identifying diseases that are strongly correlated with a patients symptoms. We show that this inability to disentangle correlation from causation can result in sub-optimal or dangerous diagnoses. To overcome this, we reformulate diagnosis as a counterfactual inference task and derive counterfactual diagnostic algorithms. We compare our counterfactual algorithms to the standard associative algorithm and 44 doctors using a test set of clinical vignettes. While the associative algorithm achieves an accuracy placing in the top 48% of doctors in our cohort, our counterfactual algorithm places in the top 25% of doctors, achieving expert clinical accuracy. Our results show that causal reasoning is a vital missing ingredient for applying machine learning to medical diagnosis. In medical diagnosis a doctor aims to explain a patient’s symptoms by determining the diseases causing them, while existing diagnostic algorithms are purely associative. Here, the authors reformulate diagnosis as a counterfactual inference task and derive new counterfactual diagnostic algorithms.

Data-driven fault diagnosis is complicated by unknown fault classes and limited training data from different fault realizations. In these situations, conventional multi-class classification approaches are not suitable for fault diagnosis. One solution is the use of anomaly classifiers that are trained using only nominal data. Anomaly classifiers can be used to detect when a fault occurs but give little information about its root cause. Hybrid fault diagnosis methods combining physically-based models and available training data have shown promising results to improve fault classification performance and identify unknown fault classes. Residual generation using grey-box recurrent neural networks can be used for anomaly classification where physical insights about the monitored system are incorporated into the design of the machine learning algorithm. In this work, an automated residual design is developed using a bipartite graph representation of the system model to design grey-box recurrent neural networks and evaluated using a real industrial case study. Data from an internal combustion engine test bench is used to illustrate the potentials of combining machine learning and model-based fault diagnosis techniques.

Decision tree optimization is notoriously difficult from a computational perspective but essential for the field of interpretable machine learning. Despite efforts over the past 40 years, only recently have optimization breakthroughs been made that have allowed practical algorithms to find optimal decision trees. These new techniques have the potential to trigger a paradigm shift where it is possible to construct sparse decision trees to efficiently optimize a variety of objective functions without relying on greedy splitting and pruning heuristics that often lead to suboptimal solutions. The contribution in this work is to provide a general framework for decision tree optimization that addresses the two significant open problems in the area: treatment of imbalanced data and fully optimizing over continuous variables. We present techniques that produce optimal decision trees over a variety of objectives including F-score, AUC, and partial area under the ROC convex hull. We also introduce a scalable algorithm that produces provably optimal results in the presence of continuous variables and speeds up decision tree construction by several orders of magnitude relative to the state-of-the art.

A structural causal model is made of endogenous (manifest) and exogenous (latent) variables. We show that endogenous observations induce linear constraints on the probabilities of the exogenous variables. This allows to exactly map a causal model into a credal network. Causal inferences, such as interventions and counterfactuals, can consequently be obtained by standard algorithms for the updating of credal nets. These natively return sharp values in the identifiable case, while intervals corresponding to the exact bounds are produced for unidentifiable queries. A characterization of the causal models that allow the map above to be compactly derived is given, along with a discussion about the scalability for general models. This contribution should be regarded as a systematic approach to represent structural causal models by credal networks and hence to systematically compute causal inferences. A number of demonstrative examples is presented to clarify our methodology. Extensive experiments show that approximate algorithms for credal networks can immediately be used to do causal inference in real-size problems.

In this paper, we study a novel problem: "automatic prescription recommendation for PD patients." To realize this goal, we first build a dataset by collecting 1) symptoms of PD patients, and 2) their prescription drug provided by neurologists. Then, we build a novel computer-aided prescription model by learning the relation between observed symptoms and prescription drug. Finally, for the new coming patients, we could recommend (predict) suitable prescription drug on their observed symptoms by our prescription model. From the methodology part, our proposed model, namely Prescription viA Learning lAtent Symptoms (PALAS), could recommend prescription using the multi-modality representation of the data. In PALAS, a latent symptom space is learned to better model the relationship between symptoms and prescription drug, as there is a large semantic gap between them. Moreover, we present an efficient alternating optimization method for PALAS. We evaluated our method using the data collected from 136 PD patients at Nanjing Brain Hospital, which can be regarded as a large dataset in PD research community. The experimental results demonstrate the effectiveness and clinical potential of our method in this recommendation task, if compared with other competing methods.

A general purpose method to detect statistical dependence, or correlation, between random variables has invaluable uses in a wide array of sciences and applications (Li, 2000; Martínez-Gómez et al., 2014; Mahdi et al., 2012). Linear correlation (Pearson, 1920) is one of the oldest statistical methods that are still widely used today. Though the assumption of linearity is not always realistic, the popularity of such method stems from its ease of computation, simplicity, interpretability, and high power when the assumption of linearity is satisfied. Several approaches have been proposed to quantify correlation, in the general case, for more complex relationships and under less stringent assumptions. Examples of these methods are the kernel based correlation (Hardoon et al., 2004; Chang et al., 2013), copula methods (Poczos et al., 2012), distance correlation (Székely et al., 2007; Székely and Rizzo, 2009), and discretization based mutual information (MI) (Steuer et al., 2002) methods such as the maximal information criterion (MIC) (Reshef et al., 2011). Issues that can be lacking in some of the existing methods include: low statistical power, high computation demand, lack of intuitive interpretability, or lack of a known distribution of the coefficient under independence that would enable computing a statistical confidence. More thorough details on the pros and cons of those methods and others can be found in several studies (de Siqueira Santos et al., 2014; N. Reshef et al., 2018).