Higher order Matching Pursuit for Low Rank Tensor Learning

Low rank tensor learning, such as tensor completion and multilinear multitask learning, has received much attention in recent years. In this paper, we propose higher order matching pursuit for low rank tensor learning problems with a convex or a nonconvex cost function, which is a generalization of the matching pursuit type methods. At each iteration, the main cost of the proposed methods is only to compute a rank-one tensor, which can be done efficiently, making the proposed methods scalable to large scale problems. Moreover, storing the resulting rank-one tensors is of low storage requirement, which can help to break the curse of dimensionality. The linear convergence rate of the proposed methods is established in various circumstances. Along with the main methods, we also provide a method of low computational complexity for approximately computing the rank-one tensors, with provable approximation ratio, which helps to improve the efficiency of the main methods and to analyze the convergence rate. Experimental results on synthetic as well as real datasets verify the efficiency and effectiveness of the proposed methods. Tensors, appearing as the higher order generalization of vectors and matrices, make it possible to represent data that have intrinsically many dimensions, and give a better understanding of the relationship behind the information from a higher order perspective. In many machine learning problems such as tensor completion [1]-[4], multilinear multitask learning (MLMTL) [5]-[7] and tensor regression [8], one often aims at learning a tensor that has low rankness. For example, in tensor completion, the goal is to learn a low rank tensor provided that only partial observations are available. In the context of MLMTL, to allow for common information shared between tasks to pursuit better generalization, by learning several tasks simultaneously, where each task is indexed by more than two indices, all the tasks can be represented by a tensor assumed to lie in a low dimensional spaces. In tensor regression, to better understand the information behind high dimensionality data, the weight vector is represented by a low rank tensor. These applications give rise to low rank tensor learning. Commonly speaking, to learn a low rank tensor, tensor learning minimizes a real-valued cost functionF: T R subject to some constraints or with regularizations to encourage the low rank property of the learned tensor.