Deep Learning
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency.
A Preliminaries on Transformers
Perplexity is a widely used metric for evaluating the performance of autoregressive language models. This metric encapsulates how well the model can predict a word.
LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models
The Transformer architecture is ubiquitously used as the building block of large-scale autoregressive language models.
Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability
We focus on the blackbox transfer-based adversarial threat model for DNN image classifiers. In the standard case, blackbox means the attacker does not have access to the gradients of the target model and makes no assumptions about its architecture.
Single-phase deep learning in cortico-cortical networks
The error-backpropagation (backprop) algorithm remains the most common solution to the credit assignment problem in artificial neural networks.
SSAL: Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection
We study adapting trained object detectors to unseen domains manifesting significant variations of object appearance, viewpoints and backgrounds.