Deep Learning
DABS 2.0: Improved Datasets and Algorithms for Universal Self Supervision Alex T amkin
Universal self-supervised learning (SSL) algorithms hold enormous promise for making machine learning accessible to high-impact domains such as protein biology, manufacturing, and genomics. We present DABS 2.0: a set of improved
0 2 0 3 1 3 1 2 drop 0 2 0 3 1 3 1 2 grow 0 2 2 0 1 2 3 0 1 2 3 shuffle 0 2 1 3 0 1 2 3 blocking 0 2 0 2 1 3 1 2 Before After active zero removed added
Sparse training is a popular technique to reduce the overhead of training large models.
Zeroth-Order Negative Curvature Finding: Escaping Saddle Points without Gradients
A recent line of work shows that, by adding uniform random perturbations, first-order (FO) methods can efficiently escape saddle points and converge to SOSP .
Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy
Large convolutional neural networks (CNN) can be difficult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping.
Algorithm 1 Learning the external stimulus s Require: (x
Figure taken and adapted from [38]. Different works consider different properties. Compared to backpropagation (BP), predictive coding (PC) allows for more flexibility in the definition, training, and evaluation of the model. The experiments reported in this paper show the best results achieved on each specific task and, as a consequence, only the effects of a specific set of hyperparameters. Feedforward networks (left) simply overfit (i.e., reproduce without performing any modification) the input samples, despite being unrelated to the training data.
Sequencer: Deep LSTM for Image Classification Y uki Tatsunami
LSTM is decomposed into vertical and horizontal LSTMs to enhance performance.