Understanding the performance of learning algorithms under information constraints is a fundamental research direction in machine learning. While performance notions such as regret in online learning have been well explored, a recent line of work explores additional constraints in learning, with a particular emphasis on limited memory [Sha14, WS19, MSSV22] (see also Section 3). In this paper, we focus on the online learning with experts problem, a general framework for sequential decision making, with memory constraints. In the online learning with experts problem, an algorithm must make predictions about the outcome of an event for T consecutive days based on the predictions of n experts. The predictions of the algorithm at a time t T can only depend on the information it has received in the previous days as well as the predictions of the experts for day t. After predictions are made, the true outcome is revealed and the algorithm and all experts receive some loss (likely depending on the accuracy of their predictions). In addition to the fact that the online experts problem has found numerous algorithmic applications [AHK12], studying the problem with memory constraints is especially interesting in light of the fact that existing algorithms explicitly track the cumulative loss of every expert and follow the advice of a leading expert, which requires Ω(n) memory. Motivated by this lack of understanding, the online learning with experts problem with memory constraints was recently introduced in [SWXZ22], which studied the case where the losses of the experts form an i.i.d.

Recent work shows that the expressive power of Graph Neural Networks (GNNs) in distinguishing non-isomorphic graphs is exactly the same as that of the Weisfeiler-Lehman (WL) graph test. In particular, they show that the WL test can be simulated by GNNs. However, those simulations involve neural networks for the 'combine' function of size polynomial or even exponential in the number of graph nodes $n$, as well as feature vectors of length linear in $n$. We present an improved simulation of the WL test on GNNs with \emph{exponentially} lower complexity. In particular, the neural network implementing the combine function in each node has only a polylogarithmic number of parameters in $n$, and the feature vectors exchanged by the nodes of GNN consists of only $O(\log n)$ bits. We also give logarithmic lower bounds for the feature vector length and the size of the neural networks, showing the (near)-optimality of our construction.

Kernel matrices, as well as weighted graphs represented by them, are ubiquitous objects in machine learning, statistics and other related fields. The main drawback of using kernel methods (learning and inference using kernel matrices) is efficiency -- given $n$ input points, most kernel-based algorithms need to materialize the full $n \times n$ kernel matrix before performing any subsequent computation, thus incurring $\Omega(n^2)$ runtime. Breaking this quadratic barrier for various problems has therefore, been a subject of extensive research efforts. We break the quadratic barrier and obtain $\textit{subquadratic}$ time algorithms for several fundamental linear-algebraic and graph processing primitives, including approximating the top eigenvalue and eigenvector, spectral sparsification, solving linear systems, local clustering, low-rank approximation, arboricity estimation and counting weighted triangles. We build on the recent Kernel Density Estimation framework, which (after preprocessing in time subquadratic in $n$) can return estimates of row/column sums of the kernel matrix. In particular, we develop efficient reductions from $\textit{weighted vertex}$ and $\textit{weighted edge sampling}$ on kernel graphs, $\textit{simulating random walks}$ on kernel graphs, and $\textit{importance sampling}$ on matrices to Kernel Density Estimation and show that we can generate samples from these distributions in $\textit{sublinear}$ (in the support of the distribution) time. Our reductions are the central ingredient in each of our applications and we believe they may be of independent interest. We empirically demonstrate the efficacy of our algorithms on low-rank approximation (LRA) and spectral sparsification, where we observe a $\textbf{9x}$ decrease in the number of kernel evaluations over baselines for LRA and a $\textbf{41x}$ reduction in the graph size for spectral sparsification.