Statistical Learning
Convergence and Alignment of Gradient Descent with Random Backpropagation Weights Ganlin Song Ruitu Xu John Lafferty Department of Statistics and Data Science
Stochastic gradient descent with backpropagation is the workhorse of artificial neural networks. It has long been recognized that backpropagation fails to be a biologically plausible algorithm. Fundamentally, it is a non-local procedure-- updating one neuron's synaptic weights requires knowledge of synaptic weights or receptive fields of downstream neurons. This limits the use of artificial neural networks as a tool for understanding the biological principles of information processing in the brain. Lillicrap et al. (2016) propose a more biologically plausible "feedback alignment" algorithm that uses random and fixed backpropagation weights, and show promising simulations. In this paper we study the mathematical properties of the feedback alignment procedure by analyzing convergence and alignment for two-layer networks under squared error loss. In the overparameter-ized setting, we prove that the error converges to zero exponentially fast, and also that regularization is necessary in order for the parameters to become aligned with the random backpropagation weights. Simulations are given that are consistent with this analysis and suggest further generalizations. These results contribute to our understanding of how biologically plausible algorithms might carry out weight learning in a manner different from Hebbian learning, with performance that is comparable with the full non-local backpropagation algorithm.
Alleviating the Sample Selection Bias in Few-shot Learning by Removing Projection to the Centroid
Despite the emergence of a number of few-shot learning methods, the sample selection bias problem, i.e., the sensitivity to the limited amount of support data, has not been well understood.
Hybrid Models for Learning to Branch
In this work, we ask two key questions. First, in a more realistic setting where only a CPU is available, is the GNN model still competitive?
Supplementary Material of " Algorithmic Stability and Generalization of an Unsupervised Feature Selection Algorithm "
Correspondence should be addressed to: qiang.cheng@uky.edu. The architecture of our algorithm is shown in Figure 1. For the training based on Eq. (2) of the main text, in each iteration of backpropagation, After training, only the trained selector is used to select features and do reconstruction during testing time. In Eq. (2) of the main text, the second term helps obtain During testing time, only the trained sub-NN is used to select features and do reconstruction. It has 5, 744 samples and 561 features.
Algorithmic Stability and Generalization of an Unsupervised Feature Selection Algorithm
Algorithmic stability is a key characteristic of an algorithm regarding its sensitivity to perturbations of input samples. In this paper, we propose an innovative unsupervised feature selection algorithm attaining this stability with provable guarantees.
Neuron with Steady Response Leads to Better Generalization Qiang Fu
Existing works have already proposed various inductive biases from diverse perspectives.
Ordered Subgraph Aggregation Networks Chendi Qian
Graph-structured data are ubiquitous across application domains ranging from chemo-and bioinformat-ics [Barabasi and Oltvai, 2004, Jumper et al., 2021, Stokes et al., 2020] to image [Simonovsky and
Nonuniform Negative Sampling and Log Odds Correction with Rare Events Data
We investigate the issue of parameter estimation with nonuniform negative sampling for imbalanced data.