We consider the training process of a neural network as a dynamical system acting on the high-dimensional weight space. Each epoch is an application of the map induced by the optimization algorithm and the loss function. Using this induced map, we can apply observables on the weight space and measure their evolution. The evolution of the observables are given by the Koopman operator associated with the induced dynamical system. We use the spectrum and modes of the Koopman operator to realize the above objectives. Our methods can help to, a priori, determine the network depth; determine if we have a bad initialization of the network weights, allowing a restart before training too long; speeding up the training time. Additionally, our methods help enable noise rejection and improve robustness. We show how the Koopman spectrum can be used to determine the number of layers required for the architecture. Additionally, we show how we can elucidate the convergence versus non-convergence of the training process by monitoring the spectrum, in particular, how the existence of eigenvalues clustering around 1 determines when to terminate the learning process. We also show how using Koopman modes we can selectively prune the network to speed up the training procedure. Finally, we show that incorporating loss functions based on negative Sobolev norms can allow for the reconstruction of a multi-scale signal polluted by very large amounts of noise.

Bayesian meta-learning enables robust and fast adaptation to new tasks with uncertainty assessment. The key idea behind Bayesian meta-learning is empirical Bayes inference of hierarchical model. In this work, we extend this framework to include a variety of existing methods, before proposing our variant based on gradient-EM algorithm. Our method improves computational efficiency by avoiding back-propagation computation in the meta-update step, which is exhausting for deep neural networks. Furthermore, it provides flexibility to the inner-update optimization procedure by decoupling it from meta-update. Experiments on sinusoidal regression, few-shot image classification, and policy-based reinforcement learning show that our method not only achieves better accuracy with less computation cost, but is also more robust to uncertainty.

We propose a fully generative model where the latent variable respects both the distances and the topology of the modeled data. The model leverages the Riemannian geometry of the generated manifold to endow the latent space with a well-defined stochastic distance measure, which is modeled as Nakagami distributions. These stochastic distances are sought to be as similar as possible to observed distances along a neighborhood graph through a censoring process. The model is inferred by variational inference and is therefore fully generative. We demonstrate how the new model can encode invariances in the learned manifolds.

We propose an accelerated-gradient-based MCMC method. It relies on a modification of the Nesterov's accelerated gradient method for strongly convex functions (NAG-SC): We first reformulate NAG-SC as a Hessian-Free High-Resolution ODE, then release the high-resolution coefficient as a free hyperparameter, and finally inject appropriate noise and discretize the diffusion process. Accelerated sampling enabled by this new hyperparameter is not only experimentally demonstrated on several learning tasks, but also theoretically quantified, both at the continuous level and after discretization. For (not-necessarily-strongly-) convex and $L$-smooth potentials, exponential convergence in $\chi^2$ divergence is proved, with a rate analogous to state-of-the-art results of underdamped Langevin dynamics, plus an additional acceleration. At the same time, the method also works for nonconvex potentials, for which we also establish exponential convergence as long as the potential satisfies a Poincar\'e inequality.

We study without-replacement SGD for solving finite-sum optimization problems. Specifically, depending on how the indices of the finite-sum are shuffled, we consider the RandomShuffle (shuffle at the beginning of each epoch) and SingleShuffle (shuffle only once) algorithms. First, we establish minimax optimal convergence rates of these algorithms up to poly-log factors. Notably, our analysis is general enough to cover gradient dominated nonconvex costs, and does not rely on the convexity of individual component functions unlike existing optimal convergence results. Secondly, assuming convexity of the individual components, we further sharpen the tight convergence results for RandomShuffle by removing the drawbacks common to all prior arts: large number of epochs required for the results to hold, and extra poly-log factor gaps to the lower bound.

Machine learning has achieved remarkable results in recent years due to the increase in the number of data and the development of computational resources. However, despite such excellent performance, machine learning models often suffer from the problem of over-fitting. Many data augmentation methods have been proposed to tackle such a problem, and one of them is called Mixup. Mixup is a recently proposed regularization procedure, which linearly interpolates a random pair of training examples. This regularization method works very well experimentally, but its theoretical guarantee is not fully discussed. In this study, we aim to find out why Mixup works well from the aspect of computational learning theory. In addition, we reveal how the effect of Mixup changes in each situation. Furthermore, we also investigated the effects of changes in the Mixup's parameter. This contributes to the search for the optimal parameters and to estimate the effects of the parameters currently used. The results of this study provide a theoretical clarification of when and how effective regularization by Mixup is.

Zeroth-order (ZO) optimization is a subset of gradient-free optimization that emerges in many signal processing and machine learning applications. It is used for solving optimization problems similarly to gradient-based methods. However, it does not require the gradient, using only function evaluations. Specifically, ZO optimization iteratively performs three major steps: gradient estimation, descent direction computation, and solution update. In this paper, we provide a comprehensive review of ZO optimization, with an emphasis on showing the underlying intuition, optimization principles and recent advances in convergence analysis. Moreover, we demonstrate promising applications of ZO optimization, such as evaluating robustness and generating explanations from black-box deep learning models, and efficient online sensor management.

The company's Stylish True Wireless is my go-to IEH for the gym and walks around the neighborhood, and I gave the Triple Driver IEH a rating of 5--the only such rating I've given to date--in my review of three wired models from 1More. So, it should come as no surprise that I was excited to review 1More's latest offering, the Dual Driver ANC Pro wireless IEH. Its form factor is similar to several of the company's models but different than the other wireless IEHs I've tried. I really like the form factor, but sound quality is most important, and the Dual Driver ANC Pro is something of a split personality in that regard. The Dual Driver ANC Pro consists of a flexible, silicone-covered neckband with thin wires emerging from both ends that connect to the earpieces. The back of the earpieces are magnetic, so they clasp together to form a necklace for convenient, safe storage when not in use.

GitHub recently announced its partnership with Hack Club to support students with coding. Committing to a $50K hardware fund, it announced working globally alongside Arduino and Adafruit on delivering the hardware tools directly to students' homes. Designed for teenagers aged 13-18, the program is free of cost wherein students will have access to hardware on a needs basis and will have guidance from the industry mentors. This clearly tells us about the importance that global companies are laying on students and kids for coding. In this article, we list 5 institutes and initiatives that are working on teaching kids coding.

Much proselytizing has occurred regarding the value and future of artificial intelligence (AI) and machine learning in healthcare. As with blockchain technology, which continues to evolve in the healthcare marketplace, AI and machine learning are constructs that require a bit of near-term expectation management. While their efficacy and value will improve with time, they are not the magic bullet (at present) that will answer the myriad care and cost delivery questions surrounding healthcare in the United States. Owing to space constraints this column is an overly simplistic contemplation of AI. As prologue to this article, I am not an AI programmer, don't play in Python, and have never built a machine learning algorithm. That said, I do have 30 years of practical experience in the healthcare trenches and have dealt with information technology (IT) systems and applications in that time, such as culling quality data and outcomes from electronic medical record (EMR) systems and deploying rudimentary analytics.