Contrastive estimation reveals topic posterior information to linear models

Using unlabeled data to find useful embeddings is a central challenge in the field of representation learning. Classical approaches to this task often start by fitting some type of structure to the unlabeled data, such as a generative model or a dictionary, and then embed future data by performing inference using the fitted structure (Blei et al., 2003; Raina et al., 2007). While this approach has sometimes enjoyed good empirical performance, it is not without its drawbacks. One issue is that learning structures and performing inference is often hard in general (Sontag and Roy, 2011; Arora et al., 2012). Another issue is that we must a priori choose a structure and method for fitting the unlabeled data, and unsupervised methods for learning these structures can be sensitive to model misspecification (Kulesza et al., 2014).