Ever since its origin in post-war research, AI has been subject to profound hyperbole, rapturous prognostications, and projected nightmares. In 2019, things have once again reached fever pitch in what science board co-chair and External Professor Melanie Mitchell wryly notes is a hype cycle that routinely ripples through her fellow computer scientists and those who fund them. Her illuminating new book, Artificial Intelligence: A Guide for Thinking Humans, lays bare the inner workings of these potent tools, exposing their realistic limits and patiently detailing our deployment errors. It is a solid history of how we got from pocket calculators to facial recognition and self-driving cars, a lucid tour of how these systems operate, and a tempered read on just how far we have to go before we're obsolete. Mitchell, a professor of computer science at Portland State University, has spent decades studying AI and writes with the measured understanding of someone who has lived on the volcano.

What do you want to find out or discover using your data? Do you have the appropriate data to analyze? Data is key to any data science and machine learning task. Data comes in different flavors such as numerical data, categorical data, text data, image data, sound data, and video data. The predictive power of a model depends on the quality of data used in building the model.

I have read a book or some posts on machine learning. I have watched some of the Coursera machine learning course. I still don't know how to get started… How do you get started in machine learning? The most common question I'm asked by developers on my newsletter is: How do I get started in machine learning? I honestly cannot remember how many times I have answered it. In this post, I lay out all of my very best thinking on this topic. You are a developer and you're interested in getting into machine learning. You read some blog posts.

Are you interested in the field of Deep Learning? Here is the short and useful Review of Deep Learning A-Z Hands-On Artificial Neural Networks. If you are in the intermediate level people who know the basics of Deep Learning and Machine Learning, including the classical algorithms like linear regression or logistic regression and more advanced topics like Artificial Neural Networks, but who want to learn more about it and explore all the different fields of Deep Learning. This is one of the Best Seller courses on Udemy where students enrolled more than 157K with 21K reviews and 4.5 average star rating. With this top-selling Deep Learning tutorial, you will learn how to create Deep Learning Algorithms in Python from two Machine Learning & Data Science experts.

Every weekday, this feed brings you our latest talks in audio format. Hear thought-provoking ideas on every subject imaginable -- from Artificial Intelligence to Zoology, and everything in between -- given by the world's leading thinkers and doers. This collection of talks, given at TED and TEDx conferences around the globe, is also available in video format.

A BSTRACT To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). First, we prove that when the gradient is zero (or bounded above by a small constant) at every data point in training, a situation called interpolation setting, there exists many-to-one alignment between student and teacher nodes in the lowest layer under mild conditions. This suggests that generalization in unseen dataset is achievable, even the same condition often leads to zero training error. Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of training. As a result, it could take a long time to converge into these small-gradient critical points. Our analysis shows that over-parameterization plays two roles: (1) it is a necessary condition for alignment to happen at the critical points, and (2) in training dynamics, it helps student nodes cover more teacher nodes with fewer iterations. Although networks with even one-hidden layer can fit any function (Hornik et al., 1989), it remains an open question how such networks can generalize to new data. Different from what traditional machine learning theory predicts, empirical evidence (Zhang et al., 2017) shows more parameters in neural network lead to better generalization. How over-parameterization yields strong generalization is an important question for understanding how deep learning works. In this paper, we analyze multi-layer ReLU networks by adopting teacher-student setting. The fixed teacher network provides the output for the student to learn via SGD. The student is over-parameterized (or over-realized): it has more nodes than the teacher. Therefore, there exists student weights whose gradient at every data point is zero. Here, we want to study the inverse problem: With small gradient at every training sample, can the student weights recover the teachers'? If so, then the generalization performance can be guaranteed if the training converges to such critical points. In this paper, we show that this so-called interpolation setting (Ma et al., 2017; Liu & Belkin, 2018; Bassily et al., 2018) leads to alignment: under certain conditions, each teacher node is provably aligned with at least one student node in the lowest layer. The condition is simply that the teacher node is observed by at least one student node, i.e., teacher's ReLU boundary lies in the activation region of that student. Therefore, more over-parameterization increases the probability of teachers being observed and thus being aligned. Furthermore, in 2-layer case, those student nodes that are not aligned with any teacher have zero contribution to the output and can be pruned.

Mini-batch stochastic gradient descent (SGD) approximates the gradient of an objective function with the average gradient of some batch of constant size. While small batch sizes can yield high-variance gradient estimates that prevent the model from learning a good model, large batches may require more data and computational effort. This work presents a method to change the batch size adaptively with model quality. We show that our method requires the same number of model updates as full-batch gradient descent while requiring the same total number of gradient computations as SGD. While this method requires evaluating the objective function, we present a passive approximation that eliminates this constraint and improves computational efficiency. We provide extensive experiments illustrating that our methods require far fewer model updates without increasing the total amount of computation.

Daniel Jiwoong Im 1, Yibo Jiang 2, and Nakul Verma 3 1 Janelia Research Campus, HHMI, Virgina 2 Harvard University, Massachusetts 3 Columbia University, New York Abstract Meta learning has emerged as an important framework for learning new tasks from just a few examples. The success of any meta-learning model depends on (i) its fast adaptation to new tasks, as well as (ii) having a shared representation across similar tasks. Here we extend the model-agnostic meta-learning (MAML) framework introduced by Finn et al. (2017) to achieve improved performance by analyzing the temporal dynamics of the optimization procedure via the Runge-Kutta method. This method enables us to gain fine-grained control over the optimization and helps us achieve both the adaptation and representation goals across tasks. By leveraging this refined control, we demonstrate that there are multiple principled ways to update MAML and show that the classic MAML optimization is simply a special case of second order Runge-Kutta method that mainly focuses on fast-adaptation. Experiments on benchmark classification, regression and reinforcement learning tasks show that this refined control helps attain improved results. 1 Introduction Building an intelligent system that can learn quickly on a new task with few examples or few experiences is one of the central goals of machine learning. Achieving this goal requires an agent that learns continuously while having the ability to adapt to new tasks with limited data. Meta-learning (Biggs, 1985) has emerged as a compelling framework that strives to attain this challenging goal. There are two main approaches to meta-learning: learning-to-optimize and learning-to-initialize the meta-model (usually encoded as deep network).

Selecting hyperparameters for unsupervised learning problems is difficult in general due to the lack of ground truth for validation. However, this issue is prevalent in machine learning, especially in clustering problems with examples including the Lagrange multipliers of penalty terms in semidefinite programming (SDP) relaxations and the bandwidths used for constructing kernel similarity matrices for Spectral Clustering. Despite this, there are not many provable algorithms for tuning these hyperparameters. In this paper, we provide a unified framework with provable guarantees for the above class of problems. We demonstrate our method on two distinct models. First, we show how to tune the hyperparameters in widely used SDP algorithms for community detection in networks. In this case, our method can also be used for model selection. Second, we show the same framework works for choosing the bandwidth for the kernel similarity matrix in Spectral Clustering for subgaussian mixtures under suitable model specification. In a variety of simulation experiments, we show that our framework outperforms other widely used tuning procedures in a broad range of parameter settings.

We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the MNIST dataset.