A general method for regularizing tensor decomposition methods via pseudo-data

Tensor decomposition methods (TDMs) have recently gained popularity as ways of performing inference for latent variable models [Anandkumar et al., 2014]. The interest in these methods is motivated by the fact that they come with theoretical global convergence guarantees in the limit of infinite data [Anandkumar et al., 2012, Arora et al., 2013]. However, a main limitation of these methods is that they lack natural methods for regularization or encouraging desired properties on the model parameters when the amount of data is limited. Previous works attempted to alleviate this drawback by modifying existing tensor decomposition methods to incorporate specific constraints, such as sparsity [Sun et al., 2015], or incorporate modeling assumptions, such as the existence of anchor words [Arora et al., 2013, Nguyen et al., 2014]. All of these works develop bespoke algorithms tailored to those constraints or assumptions. Furthermore, many of these methods impose hard constraints on the learned model, which may be detrimental as the size of the data grow--framed in the context of Bayesian intuition, when we have a lot of data, we want our methods to allow the evidence to overwhelm our priors. We introduce an alternative approach which can be applied to encourage any (differentiable) desired structure or properties on the model parameters, and which will only encourage this "prior" information when the data is insufficient. Specifically, we adopt the common view of Bayesian priors as representing "pseudo-observations" of artificial data which bias our learned model parameters towards our prior belief [Bishop, 2006]. We apply the tensor decomposition method of Anandkumar et al.