In my last post I promised to use some real-world use cases from the recent Verizon Data Breach Digest report to illustrate potential ways that machine learning be can used to detect or prevent similar incidents. For my first example, I've chosen the case of a manufacturer whose designs for an innovative new model of heavy construction equipment were stolen following a social engineering attack. They were tipped off when a primary competitor, located on another continent, introduced a new piece of equipment that looked like an exact copy of a model recently developed by the victim company. To paraphrase the Verizon report, it went like this. The threat actors identified an employee who they suspected would have access to new product design they were after -- the chief design engineer.

ICU mortality scoring systems attempt to predict patient mortality using predictive models with various clinical predictors. Examples of such systems are APACHE, SAPS and MPM. However, most such scoring systems do not actively look for and include interaction terms, despite physicians intuitively taking such interactions into account when making a diagnosis. One barrier to including such terms in predictive models is the difficulty of using most variable selection methods in high-dimensional datasets. A genetic algorithm framework for variable selection with logistic regression models is used to search for two-way interaction terms in a clinical dataset of adult ICU patients, with separate models being built for each category of diagnosis upon admittance to the ICU. The models had good discrimination across all categories, with a weighted average AUC of 0.84 (>0.90 for several categories) and the genetic algorithm was able to find several significant interaction terms, which may be able to provide greater insight into mortality prediction for health practitioners. The GA selected models had improved performance against stepwise selection and random forest models, and provides greater flexibility in terms of variable selection by being able to optimize over any modeler-defined model performance metric instead of a specific variable importance metric.

By using machine learning algorithms, we are increasingly able to use computers to perform intellectual tasks at a level approaching that of humans. Given that computers cost less than employees, many people are afraid that humans will therefore necessarily lose their jobs to computers. Contrary to this belief, in this article I show that even when a computer can perform a task more economically than a human, careful analysis suggests that humans and computers working together can sometimes yield even better business outcomes than simply replacing one with the other. Specifically, I show how a classifier with a reject option can increase worker productivity for certain types of tasks, and I show how to construct and tune such a classifier from a simple scoring function by using two thresholds. I begin with a parable featuring the same characters as the one from Part 1 of this Machine Learning Meets Economics series.

In the ongoing battle against cyber attacks, a man-machine collaboration could offer a new path to security. To keep up with cyber threats, the cybersecurity industry has turned to assistance from unsupervised artificial intelligence systems that operate independently from human analysts. But the Computer Science and Artificial Intelligence Laboratory (CSAIL) at the Massachusetts Institute of Technology in Cambridge, Mass., in partnership with the machine-learning startup PatternEx, is offering a fresh approach. Their new program, AI2, draws on what humans and machines each do best: It allows human analysts to build upon the large scale pattern recognition and learning capabilities of artificial intelligence. The industry standard right now is unsupervised machine learning, CSAIL research scientist Kalyan Veeramachaneni, who helped develop the program, says in a phone interview with The Christian Science Monitor.

The detection of rare variants is important for understanding the genetic heterogeneity in mixed samples. Recently, next-generation sequencing (NGS) technologies have enabled the identification of single nucleotide variants (SNVs) in mixed samples with high resolution. Yet, the noise inherent in the biological processes involved in next-generation sequencing necessitates the use of statistical methods to identify true rare variants. We propose a novel Bayesian statistical model and a variational expectation-maximization (EM) algorithm to estimate non-reference allele frequency (NRAF) and identify SNVs in heterogeneous cell populations. We demonstrate that our variational EM algorithm has comparable sensitivity and specificity compared with a Markov Chain Monte Carlo (MCMC) sampling inference algorithm, and is more computationally efficient on tests of low coverage ($27\times$ and $298\times$) data. Furthermore, we show that our model with a variational EM inference algorithm has higher specificity than many state-of-the-art algorithms. In an analysis of a directed evolution longitudinal yeast data set, we are able to identify a time-series trend in non-reference allele frequency and detect novel variants that have not yet been reported. Our model also detects the emergence of a beneficial variant earlier than was previously shown, and a pair of concomitant variants.

ROC curves can also be constructed from clinical prediction rules. The graphs at right come from a study of how clinical findings predict strep throat (Wigton RS, Connor JL, Centor RM. In that study, the presence of tonsillar exudate, fever, adenopathy and the absence of cough all predicted strep. The curves were constructed by computing the sensitivity and specificity of increasing numbers of clinical findings (from 0 to 4) in predicting strep. The study compared patients in Virginia and Nebraska and found that the rule performed more accurately in Virginia (area under the curve .78)

MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), together with researchers from security firm PatternEx, has revealed a new AI (Artificial Intelligence) system called AI2, which can detect 85 percent of cyber-attacks, with false positives rates five times smaller than existing solutions. The new system doesn't rely entirely on artificial intelligence (AI), but also on user input, something that researchers call analyst intuition (AI), hence its name of AI2. Researchers said they fed AI2 with over 3.6 billion lines of log files, allowing the system to scan the content with unsupervised machine-learning techniques. At the end of each day, the system presents its findings to a human operator, who then confirms or dismisses security alerts. This human feedback is then incorporated into AI2's learning system and used the next day for analyzing new logs. After their tests had concluded, MIT and PatternEx researchers said AI2 achieved an 85 percent accuracy rate in detecting cyber-attacks, which is 2.92 times better than similar automated cyber-attack detection systems used today.

MIT scientists have built a hybrid human/artificial intelligence (AI) machine that they claim can learn how to detect 85% of cyber attacks – that's roughly three times better than previous benchmarks – while reducing false positive rates by a factor of 5. Nitesh Chawla, professor of computer science at Notre Dame University, said in a statement from MIT that the machine "has the potential to become a line of defense against attacks such as fraud, service abuse and account takeover." Researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and the machine-learning startup PatternEx demonstrated the platform, called AI2, in a paper titled "AI2: Training a big data machine to defend". As the researchers describe the current state of the art, today's security systems are typically driven by either humans – so-called "analyst-driven solutions" – or by machine. The problem with security systems based on fixed rules is that they miss attacks that don't match those rules. Machine-learning approaches, as the name suggests, rely on an adaptive process that can trigger annoying numbers of false positives.

Many of those who call themselves statisticians just won't admit that data science heavily relies on and uses (heretical, rule-breaking) statistical science, or they don't recognize the true statistical nature of these data science techniques (some are 15-year old), or are opposed to the modernization of their statistical arsenal. They already missed the train when machine learning became a popular discipline (also heavily based on statistics) more than 15 years ago. Now machine learning professionals, who are statistical practitioners working on problems such as clustering, far outnumber statisticians. Many times, I have interacted with statisticians who think that anyone not calling himself statistician, knows nothing or little about statistics; see my recent bio published here, or visit the LinkedIn profiles of many data scientists, to debunk this myth. Any statistical technique that is not in their old books are considered heretical at best, or non-statistic at worst, or most of the time, not understood.

Predictive models are finding an increasing number of applications in many industries. As a result, a practical means for trading-off the cost of deploying a model versus its effectiveness is needed. Our work is motivated by risk prediction problems in healthcare. Cost-structures in domains such as healthcare are quite complex, posing a significant challenge to existing approaches. We propose a novel framework for designing cost-sensitive structured regularizers that is suitable for problems with complex cost dependencies. We draw upon a surprising connection to boolean circuits. In particular, we represent the problem costs as a multi-layer boolean circuit, and then use properties of boolean circuits to define an extended feature vector and a group regularizer that exactly captures the underlying cost structure. The resulting regularizer may then be combined with a fidelity function to perform model prediction, for example. For the challenging real-world application of risk prediction for sepsis in intensive care units, the use of our regularizer leads to models that are in harmony with the underlying cost structure and thus provide an excellent prediction accuracy versus cost tradeoff.