Review for NeurIPS paper: Beyond Lazy Training for Over-parameterized Tensor Decomposition

The main concern is about the utilization of the over-parameterization in tensor decomposition. In other words, for a rank-r tensor, tensor decomposition aims to find the r components which have physical interpretation. However, the proposed approach instead finds m O(r {2.5 \ell}) components, which could be far away from the target r components. For example, even for third-order tensor, it finds m O(r 7.5) components, much larger than r. 2. Perhaps the goal of this paper is to understand the effect of over-parameterization in tensor decomposition. However, if this is the case, the objective function is quite different to the classical one, and the algorithm is also different to simple gradient descent.