Spectral methods are popular in detecting global structures in the given data that can be represented as a matrix. However when the data matrix is sparse or noisy, classic spectral methods usually fail to work, due to localization of eigenvectors (or singular vectors) induced by the sparsity or noise. In this work, we propose a general method to solve the localization problem by learning a regularization matrix from the localized eigenvectors. Using matrix perturbation analysis, we demonstrate that the learned regularizations suppress down the eigenvalues associated with localized eigenvectors and enable us to recover the informative eigenvectors representing the global structure. We show applications of our method in several inference problems: community detection in networks, clustering from pairwise similarities, rank estimation and matrix completion problems. Using extensive experiments, we illustrate that our method solves the localization problem and works down to the theoretical detectability limits in different kinds of synthetic data. This is in contrast with existing spectral algorithms based on data matrix, non-backtracking matrix, Laplacians and those with rank-one regularizations, which perform poorly in the sparse case with noise.

This work addresses decentralized online optimization in non-stationary environments. A network of agents aim to track the minimizer of a global time-varying convex function. The minimizer evolves according to a known dynamics corrupted by an unknown, unstructured noise. At each time, the global function can be cast as a sum of a finite number of local functions, each of which is assigned to one agent in the network. Moreover, the local functions become available to agents sequentially, and agents do not have a prior knowledge of the future cost functions. Therefore, agents must communicate with each other to build an online approximation of the global function. We propose a decentralized variation of the celebrated Mirror Descent, developed by Nemirovksi and Yudin. Using the notion of Bregman divergence in lieu of Euclidean distance for projection, Mirror Descent has been shown to be a powerful tool in large-scale optimization. Our algorithm builds on Mirror Descent, while ensuring that agents perform a consensus step to follow the global function and take into account the dynamics of the global minimizer. To measure the performance of the proposed online algorithm, we compare it to its offline counterpart, where the global functions are available a priori. The gap between the two is called dynamic regret. We establish a regret bound that scales inversely in the spectral gap of the network, and more notably it represents the deviation of minimizer sequence with respect to the given dynamics. We then show that our results subsume a number of results in distributed optimization. We demonstrate the application of our method to decentralized tracking of dynamic parameters and verify the results via numerical experiments.

We deal with the efficient parallelization of Bayesian global optimization algorithms, and more specifically of those based on the expected improvement criterion and its variants. A closed form formula relying on multivariate Gaussian cumulative distribution functions is established for a generalized version of the multipoint expected improvement criterion. In turn, the latter relies on intermediate results that could be of independent interest concerning moments of truncated Gaussian vectors. The obtained expansion of the criterion enables studying its differentiability with respect to point batches and calculating the corresponding gradient in closed form. Furthermore , we derive fast numerical approximations of this gradient and propose efficient batch optimization strategies. Numerical experiments illustrate that the proposed approaches enable computational savings of between one and two order of magnitudes, hence enabling derivative-based batch-sequential acquisition function maximization to become a practically implementable and efficient standard.

Singularities of a statistical model are the elements of the model's parameter space which make the corresponding Fisher information matrix degenerate. These are the points for which estimation techniques such as the maximum likelihood estimator and standard Bayesian procedures do not admit the root-$n$ parametric rate of convergence. We propose a general framework for the identification of singularity structures of the parameter space of finite mixtures, and study the impacts of the singularity levels on minimax lower bounds and rates of convergence for the maximum likelihood estimator over a compact parameter space. Our study makes explicit the deep links between model singularities, parameter estimation convergence rates and minimax lower bounds, and the algebraic geometry of the parameter space for mixtures of continuous distributions. The theory is applied to establish concrete convergence rates of parameter estimation for finite mixture of skewnormal distributions. This rich and increasingly popular mixture model is shown to exhibit a remarkably complex range of asymptotic behaviors which have not been hitherto reported in the literature.

In a variety of application domains the content to be recommended to users is associated with text. This includes research papers, movies with associated plot summaries, news articles, blog posts, etc. Recommendation approaches based on latent factor models can be extended naturally to leverage text by employing an explicit mapping from text to factors. This enables recommendations for new, unseen content, and may generalize better, since the factors for all items are produced by a compactly-parametrized model. Previous work has used topic models or averages of word embeddings for this mapping. In this paper we present a method leveraging deep recurrent neural networks to encode the text sequence into a latent vector, specifically gated recurrent units (GRUs) trained end-to-end on the collaborative filtering task. For the task of scientific paper recommendation, this yields models with significantly higher accuracy. In cold-start scenarios, we beat the previous state-of-the-art, all of which ignore word order. Performance is further improved by multi-task learning, where the text encoder network is trained for a combination of content recommendation and item metadata prediction. This regularizes the collaborative filtering model, ameliorating the problem of sparsity of the observed rating matrix.

Interesting data often concentrate on low dimensional smooth manifolds inside a high dimensional ambient space. Random projections are a simple, powerful tool for dimensionality reduction of such data. Previous works have studied bounds on how many projections are needed to accurately preserve the geometry of these manifolds, given their intrinsic dimensionality, volume and curvature. However, such works employ definitions of volume and curvature that are inherently difficult to compute. Therefore such theory cannot be easily tested against numerical simulations to understand the tightness of the proven bounds. We instead study typical distortions arising in random projections of an ensemble of smooth Gaussian random manifolds. We find explicitly computable, approximate theoretical bounds on the number of projections required to accurately preserve the geometry of these manifolds. Our bounds, while approximate, can only be violated with a probability that is exponentially small in the ambient dimension, and therefore they hold with high probability in cases of practical interest. Moreover, unlike previous work, we test our theoretical bounds against numerical experiments on the actual geometric distortions that typically occur for random projections of random smooth manifolds. We find our bounds are tighter than previous results by several orders of magnitude.

An insistent pattern has quietly taken hold in my household. I will order some consumer product online. I will open the package, extract the thing from its protective wrappings, and retrieve the instruction manual. I will examine the product briefly, then begin to read the instruction manual. And then I will go to YouTube.

Philosophers of science are not known for agreeing with each other--contrariness is part of the job description. But for thousands of years, from Aristotle to Thomas Kuhn, those who study what science is have roughly categorized themselves into two basic camps: "realists" and "anti-realists." In philosophical terms, "anti-realists" or "empiricists" understand science as investigating the properties of observable objects via experiments. Empirical theories are constrained by the experimental results. "Realists," on the other hand, speculate more freely about the possible shape of the unobservable world, often designing mathematical explanations that cannot (yet) be tested. Isaac Newton was a realist, as are string theorists. Most scientists do not lose sleep worrying about philosophical divides. But maybe they should; Albert Einstein certainly did, as did Niels Bohr, and Erwin Schrödinger.

A recent newcomer at one of the home-education groups my family attends explained that one of the frustrations that led her to take her son out of the school system was that he wasn't being allowed to write stories. It's something he loves to do, and it seems strange that a school should obstruct that enthusiasm. But the teachers declared he wasn't ready because he can't yet write in cursive. To me this symbolizes all that is wrong with the strange obsession shared in many countries about how children learn to write. Often we teach them how to form letters based on the ones they see in their earliest reading books. And then we tell them that they must learn this hard-won skill all over again, using "joined-up" script. Yet there is no evidence that cursive has any benefits over other handwriting styles, such as manuscript, where the letters aren't joined, for the majority of children with normal development.

For decades, science fiction authors, futurists, and movie makers alike have been predicting the amazing (and sometimes catastrophic) changes that will arise with the advent of widespread artificial intelligence. So far, AI hasn't made any such crazy waves, and in many ways has quietly become ubiquitous in numerous aspects of our daily lives. From the intelligent sensors that help us take perfect pictures, to the automatic parking features in cars, to the sometimes frustrating personal assistants in smartphones, artificial intelligence of one kind of another is all around us, all the time. While we've yet to create self-aware robots like those that pepper popular movies like 2001: A Space Odyssey and Star Wars, we have made smart and often significant use of AI technology in a wide range of applications that, while not as mind-blowing as androids, still change our day-to-day lives. One place where artificial intelligence is poised to make big changes (and in some cases already is) is in education.