On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency.

The substitution effect happens when learning the stronger features insome neurons can substitute for learning these features in other neurons through updating the predictionhead.

Hence, todefend against thebroad attack methods, we propose a regularized adversarial training scheme with low computational overheads.

In this paper, we take a new look at real-world image super-resolution (real-SR) from a multi-task learning perspective.

Isomap is a nonlinear dimensionality reduction method and finds low-dimensional embedding of high-dimensional data by preserving the pairwise geodesic distances between data pointsinmanifold. In2-dimensional embedding manifoldM,thegeodesic polygonal curvePi,j canbeprojected on the straight line connected it two endpoints. Every line segment ofPi,j has a corresponding line segmentinthethestraightline. The hyper-parameters searched over include the dimension of node representation as well as hyper-parameters specific to each model.