Education
Language Models are Few-Shot Learners
Specifically, we train GPT -3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting.
Adaptive Data Augmentation on Temporal Graphs
Advanced TGNs utilize an RNN based memory module to represent a node's history as a
Online Learning for Adversaries with Memory: Price of Past Mistakes
Oren Anava, Elad Hazan, Shie Mannor
Neural Information Processing Systems http://nips.cc/
Table 3 List of key terms for reinforcement learning
Linear Regression (LR), and Support V ector Machine (SVM) are the ML algorithms in comparison. We use data from Dow Jones 30 constituent stocks to construct the environment.
A Convergent Gradient Descent Algorithm for Rank Minimization and Semidefinite Programming from Random Linear Measurements
We propose a simple, scalable, and fast gradient descent algorithm to optimize a nonconvex objective for the rank minimization problem and a closely related family of semidefinite programs.
Locally-Adaptive Nonparametric Online Learning: Supplementary Material
Next, we consider two algorithms for the problem of prediction with expert advice over trees. The following lemma (whose proof is deferred to Appendix D) formally states this fact. Suppose that Algorithm 5 is run using predictions and updates provided by Algorithm 6. Suppose that Algorithm 5 is run using predictions and updates provided by AdaNormalHedge. We start by proving a master regret bound that can be specialized to various settings of interest. Combining terms completes the proof.
Natural Neural Networks
Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, koray kavukcuoglu
We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.
Average Individual Fairness: Algorithms, Generalization and Experiments
Saeed Sharifi-Malvajerdi, Michael Kearns, Aaron Roth
The community studying fairness in machine learning has yet to settle on definitions.
A Unifying View of Optimism in Episodic Reinforcement Learning
In this paper, we provide a new framework for studying this class of algorithms. Optimistic algorithms are built upon the principle of "optimism in the face of uncertainty" (OFU).