In absence of reliable domain expertise, one might resort to inferring the latent graph structure, which is often difficult due to the vast search space of possible graphs.

Spectral inference provides fast algorithms and provable optimality for latent topic analysis. But for real data these algorithms require additional ad-hoc heuristics, and even then often produce unusable results. We explain this poor performance by casting the problem of topic inference in the framework of Joint Stochastic Matrix Factorization (JSMF) and showing that previous methods violate the theoretical conditions necessary for a good solution to exist. We then propose a novel rectification method that learns high quality topics and their interactions even on small, noisy data. This method achieves results comparable to probabilistic techniques in several domains while maintaining scalability and provable optimality.

However, deep learning as a non-linear regression tool based on statistics mostly suffers from the insufficient or non-existing training data, which is the usual case and should be overcome for autonomous learning systems.

Neural Information Processing Systems http://nips.cc/

We thank the reviewers for their thoughtful feedback. R1 questioned whether BigBiGAN truly represents an "important step forward toward answering some fundamental On R1's specific concern that we only demonstrate "good linear probe classification performance", we now have We haven't experimented with smaller datasets (e.g. R1 suggested adding comparisons to other methods, including results published after the submission deadline. Finally, R1 asked for comparisons with VQ-V AE-2 on generation and representation learning. BigBiGAN is trained unconditionally (without class information), while VQ-V AE-2 is class-conditional, so it's not VQ-V AE-2 did not report representation learning results (unsupervised or otherwise).

We study problem-dependent rates, i.e., generalization errors that scale tightly with the variance or the effective loss at the "best hypothesis."

Neural Information Processing Systems http://nips.cc/

Calculation of the log-normalizer is a major computational obstacle in applications of log-linear models with large output spaces. The problem of fast normalizer computation has therefore attracted significant attention in the theoretical and applied machine learning literature. In this paper, we analyze a recently proposed technique known as "self-normalization", which introduces a regularization term in training to penalize log normalizers for deviating from zero. This makes it possible to use unnormalized model scores as approximate probabilities. Empirical evidence suggests that self-normalization is extremely effective, but a theoretical understanding of why it should work, and how generally it can be applied, is largely lacking. We prove upper bounds on the loss in accuracy due to self-normalization, describe classes of input distributions that self-normalize easily, and construct explicit examples of high-variance input distributions. Our theoretical results make predictions about the difficulty of fitting self-normalized models to several classes of distributions, and we conclude with empirical validation of these predictions.

Neural Information Processing Systems http://nips.cc/