We consider a class of misspecified dynamical models where the governing term is only approximately known. Under the assumption that observations of the system's evolution are accessible for various initial conditions, our goal is to infer a non-parametric correction to the misspecified driving term such as to faithfully represent the system dynamics and devise system evolution predictions for unobserved initial conditions. We model the unknown correction term as a Gaussian Process and analyze the problem of efficient experimental design to find an optimal correction term under constraints such as a limited experimental budget. We suggest a novel formulation for experimental design for this Gaussian Process and show that approximately optimal (up to a constant factor) designs may be efficiently derived by utilizing results from the literature on submodular optimization. Our numerical experiments exemplify the effectiveness of these techniques.

In this paper, we propose a novel sufficient decrease technique for variance reduced stochastic gradient descent methods such as SAG, SVRG and SAGA. In order to make sufficient decrease for stochastic optimization, we design a new sufficient decrease criterion, which yields sufficient decrease versions of variance reduction algorithms such as SVRG-SD and SAGA-SD as a byproduct. We introduce a coefficient to scale current iterate and satisfy the sufficient decrease property, which takes the decisions to shrink, expand or move in the opposite direction, and then give two specific update rules of the coefficient for Lasso and ridge regression. Moreover, we analyze the convergence properties of our algorithms for strongly convex problems, which show that both of our algorithms attain linear convergence rates. We also provide the convergence guarantees of our algorithms for non-strongly convex problems. Our experimental results further verify that our algorithms achieve significantly better performance than their counterparts.

Over the last years, the world has witnessed the emergence of data sets of an unprecedented size across different scientific disciplines. The large volume of such data sets presents new challenges as gathering, storing, and analyzing them becomes expensive. In the context of millions or even billions of data points, existing proven algorithms "suddenly" become computationally infeasible while data sets may not fit on single machines anymore but must be stored on clusters of machines. As a consequence, new algorithms are required to scale to this massive data setting. While one could focus on single machine learning problems and come up with endless new algorithms, we focus on a more general approach: we investigate coresets -- succinct, small summaries of large data sets -- so that solutions found on the summary are provably competitive with solution found on the full data set.

The problem of population recovery refers to estimating a distribution based on incomplete or corrupted samples. Consider a random poll of sample size $n$ conducted on a population of individuals, where each pollee is asked to answer $d$ binary questions. We consider one of the two polling impediments: (a) in lossy population recovery, a pollee may skip each question with probability $\epsilon$, (b) in noisy population recovery, a pollee may lie on each question with probability $\epsilon$. Given $n$ lossy or noisy samples, the goal is to estimate the probabilities of all $2^d$ binary vectors simultaneously within accuracy $\delta$ with high probability. This paper settles the sample complexity of population recovery. For lossy model, the optimal sample complexity is $\tilde\Theta(\delta^{-2\max\{\frac{\epsilon}{1-\epsilon},1\}})$, improving the state of the art by Moitra and Saks in several ways: a lower bound is established, the upper bound is improved and the result depends at most on the logarithm of the dimension. Surprisingly, the sample complexity undergoes a phase transition from parametric to nonparametric rate when $\epsilon$ exceeds $1/2$. For noisy population recovery, the sharp sample complexity turns out to be more sensitive to dimension and scales as $\exp(\Theta(d^{1/3} \log^{2/3}(1/\delta)))$ except for the trivial cases of $\epsilon=0,1/2$ or $1$. For both models, our estimators simply compute the empirical mean of a certain function, which is found by pre-solving a linear program (LP). Curiously, the dual LP can be understood as Le Cam's method for lower-bounding the minimax risk, thus establishing the statistical optimality of the proposed estimators. The value of the LP is determined by complex-analytic methods.

Stochastic Gradient Langevin Dynamics (SGLD) is a popular variant of Stochastic Gradient Descent, where properly scaled isotropic Gaussian noise is added to an unbiased estimate of the gradient at each iteration. This modest change allows SGLD to escape local minima and suffices to guarantee asymptotic convergence to global minimizers for sufficiently regular non-convex objectives (Gelfand and Mitter, 1991). The present work provides a nonasymptotic analysis in the context of non-convex learning problems, giving finite-time guarantees for SGLD to find approximate minimizers of both empirical and population risks. As in the asymptotic setting, our analysis relates the discrete-time SGLD Markov chain to a continuous-time diffusion process. A new tool that drives the results is the use of weighted transportation cost inequalities to quantify the rate of convergence of SGLD to a stationary distribution in the Euclidean $2$-Wasserstein distance.

Manifold Markov chain Monte Carlo algorithms have been introduced to sample more effectively from challenging target densities exhibiting multiple modes or strong correlations. Such algorithms exploit the local geometry of the parameter space, thus enabling chains to achieve a faster convergence rate when measured in number of steps. However, often acquiring local geometric information increases computational complexity per step to the extent that sampling from high-dimensional targets becomes inefficient in terms of total computational time. This paper analyzes the computational complexity of manifold Langevin Monte Carlo and proposes a manifold adaptive Monte Carlo sampler aimed at balancing the benefits of exploiting local geometry with computational requirements to achieve a high effective sample size for a given computational cost. The suggested strategy randomly switches between a local geometric and an adaptive proposal kernel via a schedule to regulate the frequency of manifold-based updates. An exponentially decaying schedule is put forward that enables more frequent updates of geometric information in early transient phases of the chain, while saving computational time in late stationary phases. The average complexity can be manually set depending on the need for geometric exploitation posed by the underlying model.

We develop a two-defender (Alice and Bob) invasion game using the method of projective simulation as an embodied model for artificial intelligence. We hope that it will be the first step towards the effect of perception on different actions in a given game. As a given perception of a given situation, the agent, say Alice, encounters some attack symbols coming from the right attacker where she can learn to prevent. However, some of these percepts are invisible for her. Instead, she perceives some other signs that are related to her partner's (Bob) task. We elaborate an example in which an agent perceives an equal portion of percepts from both attackers. Alice can choose to concentrate on her job, though she loses some attacks. Alternatively, she can have some sort of cooperation with Bob to get and give help. It follows that the maximum blocking efficiency in concentration is just the minimum blocking efficiency in cooperation. Furthermore, Alice would have a choice to select two different forgetting factors for blocking attacks and for helping task. Therefore, she can choose between herself and the other. Consequently, selfishness is discerned as an only Nash equilibrium in this game. It is a pure strategy and Pareto optimal and containing Shapley value in this superadditive coalition. Finally, we propose another perception for the same situation that can be tracked in the future regarding the present study.

Welcome to the third installment of Learning AI if You Suck at Math. If you missed the earlier articles be sure to check out part 1, part 2, part 4, part 5, part 6 and part 7. Today we're going to build our own Deep Learning Dream Machine. This machine will slice through neural networks like a hot laser through butter. Other than forking over $129,000 for Nvidia's DGX-1, the AI supercomputer in a box, you simply can't get better performance than what I'll show you right here. Before we dig into building a DL beast, I want to give you the easiest upgrade path. If you don't want to build an entirely new machine, you still have one perfectly awesome option. Simply upgrade your GPU (with either a Titan X or a GTX 1080) and get VMware Workstation or use another virtualization software that supports GPU acceleration! Or you could simply install Ubuntu bare metal and if you need a Windows machine run that in a VM, so you max your performance for deep learning.

Scientists in Japan have developed an AI that can decode patterns in the brain to predict what a person is seeing or imagining. In a new study, researchers used signal patterns derived from a deep neural network to predict visual features from fMRI scans. Their'decoder' was able to identify objects with a high degree of accuracy, and the researchers say the breakthrough could pave the way for more advanced'brain-machine interfaces.' In a new study, researchers used signal patterns derived from a deep neural network to predict visual features from fMRI scans. Their'decoder' was able to identify objects with a high degree of accuracy.

The code used by the brain to recognise faces may finally have been cracked by a mind-reading computer. Scientists programmed an algorithm that is able to recreate images of faces shown to monkeys in astonishing detail by measuring brain activity. The findings seem to solve one of the most difficult problems in neuroscience - and researchers say the explanation is surprisingly simple. The discovery could lead to software in the future that allows machines to probe further into our minds. Researchers have made a computer algorithm that is able to accurately recreate faces seen by macaque monkeys by measuring their brain activity.