The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not encode the sequence into a fixed-size representation but instead requires to memorize all the hidden states. These two limitations restrict the use of the softmax attention mechanism to relatively small-scale applications with short sequences and few lookups per sequence. In this work we introduce a family of linear attention mechanisms designed to overcome the two limitations listed above. We show that removing the softmax non-linearity from the traditional attention formulation yields constant-time attention lookups and fixed-size representations of the attended sequences. These properties make these linear attention mechanisms particularly suitable for large-scale applications with extreme query loads, real-time requirements and memory constraints. Early experiments on a question answering task show that these linear mechanisms yield significantly better accuracy results than no attention, but obviously worse than their softmax alternative.

This paper considers a restriction to nonnegative matrix factorization in which at least one matrix factor is stochastic. That is, the elements of the matrix factors are nonnegative and the columns of one matrix factor sum to 1. This restriction includes topic models, a popular method for analyzing unstructured data. It also includes a method for storing and finding pictures. The paper presents necessary and sufficient conditions on the observed data such that the factorization is unique. In addition, the paper characterizes natural bounds on the parameters for any observed data and presents a consistent least squares estimator. The results are illustrated using a topic model analysis of PhD abstracts in economics and the problem of storing and retrieving a set of pictures of faces. The views expressed in this article are those of the author and do not necessarily reflect those of the Federal Trade Commission. I'm grateful for continued discussions about this problem with Devesh Raval and Nathan Wilson.

We consider the problem of Robust PCA in the fully and partially observed settings. Without corruptions, this is the well-known matrix completion problem. From a statistical standpoint this problem has been recently well-studied, and conditions on when recovery is possible (how many observations do we need, how many corruptions can we tolerate) via polynomial-time algorithms is by now understood. This paper presents and analyzes a non-convex optimization approach that greatly reduces the computational complexity of the above problems, compared to the best available algorithms. In particular, in the fully observed case, with $r$ denoting rank and $d$ dimension, we reduce the complexity from $\mathcal{O}(r^2d^2\log(1/\varepsilon))$ to $\mathcal{O}(rd^2\log(1/\varepsilon))$ -- a big savings when the rank is big. For the partially observed case, we show the complexity of our algorithm is no more than $\mathcal{O}(r^4d \log d \log(1/\varepsilon))$. Not only is this the best-known run-time for a provable algorithm under partial observation, but in the setting where $r$ is small compared to $d$, it also allows for near-linear-in-$d$ run-time that can be exploited in the fully-observed case as well, by simply running our algorithm on a subset of the observations.

General predictive models do not provide a measure of confidence in predictions without Bayesian assumptions. A way to circumvent potential restrictions is to use conformal methods for constructing non-parametric confidence regions, that offer guarantees regarding validity. In this paper we provide a detailed description of a computationally efficient conformal procedure for Kernel Ridge Regression (KRR), and conduct a comparative numerical study to see how well conformal regions perform against the Bayesian confidence sets. The results suggest that conformalized KRR can yield predictive confidence regions with specified coverage rate, which is essential in constructing anomaly detection systems based on predictive models.

Principal component analysis (PCA) is a technique to find orthonormal vectors, which are a linear combination of the attributes of the data, that explain the variance structure of the data [12]. Since a few orthonormal vectors usually explain most of the variance, PCA is often used to reduce dimension of the data by keeping only a few of the orthonormal vectors. These orthonormal vectors are called principal components (PCs). For dimensionality reduction, we are given target dimension p, the number of PCs. To measure accuracy, given p principal components, first, the original data is projected into the lower dimension using the PCs. Next, the projected data in the lower dimension is lifted to the original dimension using the PCs. Observe that this procedure causes loss of some information if p is smaller than the dimension of the original attribute space. The reconstruction error is defined by the difference between the projected-and-lifted data and the original data. To select the best p PCs, the following two objective functions are usually used: [P1] minimization of the reconstruction error, [P2] maximization of the variance of the projected data.

We analyze a compression scheme for large data sets that randomly keeps a small percentage of the components of each data sample. The benefit is that the output is a sparse matrix and therefore subsequent processing, such as PCA or K-means, is significantly faster, especially in a distributed-data setting. Furthermore, the sampling is single-pass and applicable to streaming data. The sampling mechanism is a variant of previous methods proposed in the literature combined with a randomized preconditioning to smooth the data. We provide guarantees for PCA in terms of the covariance matrix, and guarantees for K-means in terms of the error in the center estimators at a given step. We present numerical evidence to show both that our bounds are nearly tight and that our algorithms provide a real benefit when applied to standard test data sets, as well as providing certain benefits over related sampling approaches.

A breakthrough will allow machines to learn by observing. This Turing Learning, as its inventors have named it, promises smarter drones that could detect militants engaging in behavior that could endanger troops, like planting roadside bombs. Still in its infancy, the new machine learning technique is named for British mathematician Alan Turing, whose famous test challenges artificial intelligences to fool a human into thinking he or she is conversing with another human. In Turing learning, a program dubbed the "classifier" tries to learn about a system designed to fool it. In certain ways, Turing Learning resembles many existing machine-learning systems.

In the data science course that I instruct, we cover most of the data science pipeline but focus especially on machine learning. Besides teaching model evaluation procedures and metrics, we obviously teach the algorithms themselves, primarily for supervised learning. Near the end of this 11-week course, we spend a few hours reviewing the material that has been covered throughout the course, with the hope that students will start to construct mental connections between all of the different things they have learned. One of the skills that I want students to be able to take away from this course is the ability to intelligently choose between supervised learning algorithms when working a machine learning problem. Although there is some value in the "brute force" approach (try everything and see what works best), there is a lot more value in being able to understand the trade-offs you're making when choosing one algorithm over another.

Marketers will need to transform from campaigns to real-time, continuous interaction with customers via intelligent agents. So says Forrester Research VP and Principal Analyst Brian Hopkins, co-author (with Adam Silverman) of a new Forrester Research report, The Top Emerging Technologies to Watch: 2017 to 2021 ( 499 for individual purchase). It's about the top 15 developing technologies that will help businesses become more customer-obsessed over the next years. Forrester is obsessed with customer-obsession, which it says is essential to a modern brand and which is characterized by several key principles. According to the research firm, such a customer-focused company is led by insights from and about customers, responds quickly and is thoroughly connected everywhere. The report chose 15 technologies for their impact on companies employing these principles.

A joint UT Austin/Cornell team has taught a machine learning system based on the free/open Torch library to correctly guess the content of pixellated or blurred redactions with high accuracy: for masked faces that humans correctly guess 0.19% of the time, the system can make a correct guess 83% of the time, when given five tries. Redaction errors have plagued data-releases since the earliest days of the net; who can forget the hilarity of companies and agencies that added black boxes in an overlay to their PDFs, or left Word's document history (including all the deleted passages) intact on their sensitive releases? Or the pedophile whose twirly-faced redaction was de-twirled to catch and prosecute him? These days, the best practice seems to be opening the images in a bitmap editor, then replacing them with black squares. "We're using this off-the-shelf, poor man's approach," says Vitaly Shmatikov, co-author of the paper and professor at Cornell.