ROC curves can also be constructed from clinical prediction rules. The graphs at right come from a study of how clinical findings predict strep throat (Wigton RS, Connor JL, Centor RM. In that study, the presence of tonsillar exudate, fever, adenopathy and the absence of cough all predicted strep. The curves were constructed by computing the sensitivity and specificity of increasing numbers of clinical findings (from 0 to 4) in predicting strep. The study compared patients in Virginia and Nebraska and found that the rule performed more accurately in Virginia (area under the curve .78)

MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), together with researchers from security firm PatternEx, has revealed a new AI (Artificial Intelligence) system called AI2, which can detect 85 percent of cyber-attacks, with false positives rates five times smaller than existing solutions. The new system doesn't rely entirely on artificial intelligence (AI), but also on user input, something that researchers call analyst intuition (AI), hence its name of AI2. Researchers said they fed AI2 with over 3.6 billion lines of log files, allowing the system to scan the content with unsupervised machine-learning techniques. At the end of each day, the system presents its findings to a human operator, who then confirms or dismisses security alerts. This human feedback is then incorporated into AI2's learning system and used the next day for analyzing new logs. After their tests had concluded, MIT and PatternEx researchers said AI2 achieved an 85 percent accuracy rate in detecting cyber-attacks, which is 2.92 times better than similar automated cyber-attack detection systems used today.

MIT scientists have built a hybrid human/artificial intelligence (AI) machine that they claim can learn how to detect 85% of cyber attacks – that's roughly three times better than previous benchmarks – while reducing false positive rates by a factor of 5. Nitesh Chawla, professor of computer science at Notre Dame University, said in a statement from MIT that the machine "has the potential to become a line of defense against attacks such as fraud, service abuse and account takeover." Researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and the machine-learning startup PatternEx demonstrated the platform, called AI2, in a paper titled "AI2: Training a big data machine to defend". As the researchers describe the current state of the art, today's security systems are typically driven by either humans – so-called "analyst-driven solutions" – or by machine. The problem with security systems based on fixed rules is that they miss attacks that don't match those rules. Machine-learning approaches, as the name suggests, rely on an adaptive process that can trigger annoying numbers of false positives.

Many of those who call themselves statisticians just won't admit that data science heavily relies on and uses (heretical, rule-breaking) statistical science, or they don't recognize the true statistical nature of these data science techniques (some are 15-year old), or are opposed to the modernization of their statistical arsenal. They already missed the train when machine learning became a popular discipline (also heavily based on statistics) more than 15 years ago. Now machine learning professionals, who are statistical practitioners working on problems such as clustering, far outnumber statisticians. Many times, I have interacted with statisticians who think that anyone not calling himself statistician, knows nothing or little about statistics; see my recent bio published here, or visit the LinkedIn profiles of many data scientists, to debunk this myth. Any statistical technique that is not in their old books are considered heretical at best, or non-statistic at worst, or most of the time, not understood.

Predictive models are finding an increasing number of applications in many industries. As a result, a practical means for trading-off the cost of deploying a model versus its effectiveness is needed. Our work is motivated by risk prediction problems in healthcare. Cost-structures in domains such as healthcare are quite complex, posing a significant challenge to existing approaches. We propose a novel framework for designing cost-sensitive structured regularizers that is suitable for problems with complex cost dependencies. We draw upon a surprising connection to boolean circuits. In particular, we represent the problem costs as a multi-layer boolean circuit, and then use properties of boolean circuits to define an extended feature vector and a group regularizer that exactly captures the underlying cost structure. The resulting regularizer may then be combined with a fidelity function to perform model prediction, for example. For the challenging real-world application of risk prediction for sepsis in intensive care units, the use of our regularizer leads to models that are in harmony with the underlying cost structure and thus provide an excellent prediction accuracy versus cost tradeoff.

A new artificial intelligence (AI) system developed by MIT researchers promises to offer increased threat detection capabilities and reduce false positive rates, boosting incident response and productivity in the security world. The team, based at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), detailed in the paper AI2: Training a big data machine to defend [PDF], how the new platform achieves three times higher prediction capabilities, and is able to deliver significantly fewer false positive rates than current analytics models. The team showcased the AI2 platform last week at the IEEE International Conference on Big Data Security, and released the study to the public earlier today. The paper explains how the tool combines AI with'analyst intuition' to create a learning model whereby intermittent human analyst feedback is layered into a continuous unsupervised machine learning system. "You can think about the system as a virtual analyst," commented CSAIL research scientist Kalyan Veeramachaneni, who designed AI2 alongside PatternEx chief data scientist and former CSAIL researcher, Ignacio Arnaldo. "It continuously generates new models that it can refine in as little as a few hours, meaning it can improve its detection rates significantly and rapidly," he added.

Distant labeling for information extraction (IE) suffers from noisy training data. We describe a way of reducing the noise associated with distant IE by identifying coupling constraints between potential instance labels. As one example of coupling,items in a list are likely to have the same label.A second example of coupling comes from analysis of document structure: in some corpora,sections can be identified such that items in the same section are likely to have the same label. Such sections do not exist in all corpora, but we show that augmenting a large corpus with coupling constraints from even a small, well-structured corpus can improve performance substantially, doubling F1 on one task.

In this paper, we investigate the usage of autoencoders in modeling textual data. Traditional autoencoders suffer from at least two aspects: scalability with the high dimensionality of vocabulary size and dealing with task-irrelevant words. We address this problem by introducing supervision via the loss function of autoencoders. In particular, we first train a linear classifier on the labeled data, then define a loss for the autoencoder with the weights learned from the linear classifier. To reduce the bias brought by one single classifier, we define a posterior probability distribution on the weights of the classifier, and derive the marginalized loss of the autoencoder with Laplace approximation. We show that our choice of loss function can be rationalized from the perspective of Bregman Divergence, which justifies the soundness of our model. We evaluate the effectiveness of our model on six sentiment analysis datasets, and show that our model significantly outperforms all the competing methods with respect to classification accuracy. We also show that our model is able to take advantage of unlabeled dataset and get improved performance. We further show that our model successfully learns highly discriminative feature maps, which explains its superior performance.

Motivated by a number of security applications, among which border patrolling, we study, to the best of our knowledge, the first Security Game model in which patrolling strategies need to be combined with responses to signals raised by an alarm system, which is spatially uncertain (i.e., it is uncertain over the exact location the attack is ongoing) and is affected by false negatives (i.e., the missed detection rate of an attack may be positive). Ours is an infinite-horizon patrolling scenario on a graph, where a single patroller moves. We study the properties of the game model in terms of computational issues and form of the optimal strategies and we provide an approach to solve it. Finally, we provide an experimental analysis of our techniques.

ICU mortality risk prediction may help clinicians take effective interventions to improve patient outcome. Existing machine learning approaches often face challenges in integrating a comprehensive panel of physiologic variables and presenting to clinicians interpretable models. We aim to improve both accuracy and interpretability of prediction models by introducing Subgraph Augmented Non-negative Matrix Factorization (SANMF) on ICU physiologic time series. SANMF converts time series into a graph representation and applies frequent subgraph mining to automatically extract temporal trends. We then apply non-negative matrix factorization to group trends in a way that approximates patient pathophysiologic states. Trend groups are then used as features in training a logistic regression model for mortality risk prediction, and are also ranked according to their contribution to mortality risk. We evaluated SANMF against four empirical models on the task of predicting mortality or survival 30 days after discharge from ICU using the observed physiologic measurements between 12 and 24 hours after admission. SANMF outperforms all comparison models, and in particular, demonstrates an improvement in AUC (0.848 vs. 0.827, p<0.002) compared to a state-of-the-art machine learning method that uses manual feature engineering. Feature analysis was performed to illuminate insights and benefits of subgraph groups in mortality risk prediction.