Implicit Regularization with Polynomial Growth in Deep Tensor Factorization

Gunasekar et al. (2017) observed We study the implicit regularization effects of that for matrix factorization when there are no constraints on deep learning in tensor factorization. While implicit the rank, the solution of the optimization problem via gradient regularization in deep matrix and'shallow' descent turns out to be a low-rank matrix. Furthermore, tensor factorization via linear and certain type of they conjectured that, with small enough learning rate and non-linear neural networks promotes low-rank solutions initialization, gradient descent on full-dimensional matrix with at most quadratic growth, we show factorization converges to the solution with minimal nuclear that its effect in deep tensor factorization grows norm. Arora et al. (2019) and Razin & Cohen (2020) extended polynomially with the depth of the network. This the analysis to deep matrix factorization and showed provides a remarkably faithful description of the in this case that implicit regularization of gradient descent observed experimental behaviour. Using numerical cannot be formulated as a norm-minimization problem. By experiments, we demonstrate the benefits of studying the dynamics of gradient descent, they found theoretically this implicit regularization in yielding a more accurate and experimentally that it instead promotes sparsity estimation and better convergence properties. of the singular values of the learned matrix, indicating that implicit regularization in deep learning has to be studied from a dynamical point of view. Moreover, Razin et al. (2021) studied implicit regularization in'shallow' tensor