Optimization is a ubiquitous modeling tool and is often deployed in settings which repeatedly solve similar instances of the same problem. Amortized optimization methods use learning to predict the solutions to problems in these settings. This leverages the shared structure between similar problem instances. In this tutorial, we will discuss the key design choices behind amortized optimization, roughly categorizing 1) models into fully-amortized and semi-amortized approaches, and 2) learning methods into regression-based and objectivebased. We then view existing applications through these foundations to draw connections between them, including for manifold optimization, variational inference, sparse coding, meta-learning, control, reinforcement learning, convex optimization, and deep equilibrium networks. This framing enables us easily see, for example, that the amortized inference in variational autoencoders is conceptually identical to value gradients in control and reinforcement learning as they both use fully-amortized models with an objective-based loss.

Continual learning (CL) (Ring, 1995; Thrun, 1995) is a branch of machine learning where the model is exposed to a sequence of tasks with the hope of exploiting existing knowledge to adapt quickly to new tasks. The research in continual learning has seen a surge in the past few years with the explicit focus of developing algorithms that can alleviate catastrophic forgetting (McCloskey and Cohen, 1989)--whereby the model abruptly forgets the information of the past when trained on new tasks. While most of the research in continual learning is focused on developing learning algorithms, that can perform better than naive fine-tuning on a stream of data, the role of model architecture, to the best of our knowledge, is not explicitly studied in any of the existing works. Even the class of parameter isolation or expansion-based methods, for example (Rusu et al., 2016; Yoon et al., 2018), have a cursory focus on the model architecture insofar that they assume a specific architecture and try to find an algorithm operating on the architecture. Orthogonal to this direction for designing algorithms, our motivation is that the inductive biases induced by different architectural components are important for continual learning. We seek to characterize the implication of different architectural choices. To motivate, consider a ResNet-18 model (He et al., 2016) on Split CIFAR-100, where CIFAR-100 dataset (Krizhevsky et al., 2009) is split into 20 disjoint sets--a prevalent architecture and benchmark in the existing continual learning works. Figure 1a shows that explicitly designed CL algorithms, EWC (Kirkpatrick et al., 2017) (a parameter regularization-based method) and experience replay (Riemer et al., Work done during an internship at DeepMind.

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow networks, which tend to get stuck in badly-generalizing local minima. Here we investigate the cross-over between these two regimes in the high-dimensional setting, and in particular investigate the connection between the so-called mean-field/hydrodynamic regime and the seminal approach of Saad & Solla. Focusing on the case of Gaussian data, we study the interplay between the learning rate, the time scale, and the number of hidden units in the high-dimensional dynamics of stochastic gradient descent (SGD). Our work builds on a deterministic description of SGD in high-dimensions from statistical physics, which we extend and for which we provide rigorous convergence rates.

Getting an introduction to basic Machine Learning concepts doesn't have to be cumbersome or expensive. In fact, there is an abundance of free online blog posts, videos, and coding tutorials to walk you through the basics – from introductory content to common applications to algorithms to applied skills. This post includes a round-up of some of the best free options for an introductory look at ML. It's sequenced to logically follow through with each one, with each concept building upon the last. You can also pick and choose based on prior experience and interest. In the end, you'll find two applied-skill ML tutorials as well as a link to additional blog posts if you want to continue learning more about Machine Learning.

PyTorch is a very powerful framework for building deep learning. This framework is not as complex to learn as compared to other deep learning frameworks because of its straightforward way of model building. In this article, we will discuss how to build an end-to-end deep learning model that can be helpful for a novice machine learning practitioner. Through this tutorial, we will demonstrate how to define and use a convolutional neural network (CNN) in a very easy way by explaining each of the steps in detail. The major points to be covered in this article are listed below.

In industries from healthcare to education to finance to manufacturing, quarantine and extended work-from-home forced companies to use technology to reimagine nearly every facet of their operations. As the world reopens in fits and starts, we analyze the industries poised to thrive in a post-Covid world. As the Covid-19 pandemic has charted its unprecedented path around the world, it's carried with it the question: What will Covid-19's legacy be? From healthcare to education to entertainment to manufacturing, technology innovators are stepping forward to help answer that question. "Crisis can be… a catalyst or can speed up changes that are on the way -- it almost can serve as an accelerant." In the wake of the outbreak, everything from doctors appointments to schooling to workouts went online. As more people have worked, learned, banked, exercised, relaxed, and even sought medical care from home during Covid-19, they have gotten a crash course in just how much can be accomplished at ...

Artificial intelligence (AI) is being used to change our lives everyday. When it comes to building AI programs, there are two approaches programmers tend to choose: supervised or unsupervised machine learning. The simple distinction between these is supervised machine learning utilizes labeled data to predict outcomes, while unsupervised machine learning does not. There are, however, some differences between the two techniques, as well as critical areas where one surpasses the other. In this article, we will break down some of these differences with examples of both supervised and unsupervised learning. Understanding the differences between supervised vs unsupervised machine learning can be tricky, but we will try to sort it out starting with supervised learning.

A 2019 study conducted by DataChildFutures found that 46% of participating Italian households had AI-powered speakers, while 40% of toys were connected to the internet. More recent research suggests that by 2023 more than 275 million intelligent voice assistants, such as Amazon Echo or Google Home, will be installed in homes worldwide. As younger generations grow up interacting with AI-enabled devices, more consideration should be given to the impact of this technology on children, their rights and wellbeing. AI-powered learning tools and approaches are often regarded as critical drivers of innovation in the education sector. Often recognized for its ability to improve the quality of learning and teaching, artificial intelligence is being used to monitor students' level of knowledge and learning habits, such as rereading and task prioritization, and ultimately to provide a personalized approach to learning. Knewton is one example of AI-enabled learning software that identifies knowledge gaps and curates education content in line with user needs.

Due to the fast pace of life and online communications, the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing. Restoring diacritics and correcting spelling is important for proper language use and disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately, i.e., state-of-the-art diacritics restoration methods do not tolerate other typos. In this work, we tackle both problems at once by employing newly-developed ByT5 byte-level transformer models. Our simultaneous diacritics restoration and typos correction approach demonstrates near state-of-the-art performance in 13 languages, reaching >96% of the alpha-word accuracy. We also perform diacritics restoration alone on 12 benchmark datasets with the additional one for the Lithuanian language. The experimental investigation proves that our approach is able to achieve comparable results (>98%) to previously reported despite being trained on fewer data. Our approach is also able to restore diacritics in words not seen during training with >76% accuracy. We also show the accuracies to further improve with longer training. All this shows a great real-world application potential of our suggested methods to more data, languages, and error classes.

Differing from the well-developed horizontal object detection area whereby the computing-friendly IoU based loss is readily adopted and well fits with the detection metrics. In contrast, rotation detectors often involve a more complicated loss based on SkewIoU which is unfriendly to gradient-based training. In this paper, we argue that one effective alternative is to devise an approximate loss who can achieve trend-level alignment with SkewIoU loss instead of the strict value-level identity. Specifically, we model the objects as Gaussian distribution and adopt Kalman filter to inherently mimic the mechanism of SkewIoU by its definition, and show its alignment with the SkewIoU at trend-level. This is in contrast to recent Gaussian modeling based rotation detectors e.g. GWD, KLD that involves a human-specified distribution distance metric which requires additional hyperparameter tuning. The resulting new loss called KFIoU is easier to implement and works better compared with exact SkewIoU, thanks to its full differentiability and ability to handle the non-overlapping cases. We further extend our technique to the 3-D case which also suffers from the same issues as 2-D detection. Extensive results on various public datasets (2-D/3-D, aerial/text/face images) with different base detectors show the effectiveness of our approach.