Education
Multi-Task Learning as Multi-Objective Optimization
To this end, we use algorithms developed in the gradient-based multi-objective optimization literature. These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks. We therefore propose an upper bound for the multi-objective loss and show that it can be optimized efficiently. We further prove that optimizing this upper bound yields a Pareto optimal solution under realistic assumptions.
The Description Length of Deep Learning models
Solomonoff's general theory of inference (Solomonoff, 1964) and the Minimum Description Length principle (Grünwald, 2007; Rissanen, 2007) formalize Occam's razor, and hold that a good model of data is a model that is good at losslessly compressing the data, including the cost of describing the model itself. Deep neural networks might seem to go against this principle given the large number of parameters to be encoded. We demonstrate experimentally the ability of deep neural networks to compress the training data even when accounting for parameter encoding. The compression viewpoint originally motivated the use of variational methods in neural networks (Hinton and V an Camp, 1993; Schmidhuber, 1997). Unexpectedly, we found that these variational methods provide surprisingly poor compression bounds, despite being explicitly built to minimize such bounds. This might explain the relatively poor practical performance of variational methods in deep learning. On the other hand, simple incremental encoding methods yield excellent compression values on deep networks, vindicating Solomonoff's approach.
Diverse Ensemble Evolution: Curriculum Data-Model Marriage
Tianyi Zhou, Shengjie Wang, Jeff A. Bilmes
Neural Information Processing Systems http://nips.cc/
Lifelong Inverse Reinforcement Learning
Jorge Mendez, Shashank Shivkumar, Eric Eaton
Neural Information Processing Systems http://nips.cc/
Disconnected Manifold Learning for Generative Adversarial Networks
Mahyar Khayatkhoei, Maneesh K. Singh, Ahmed Elgammal
Next, we show how using a collection of generators can address this problem, providing new insights into the success of such multi-generator GANs.
DeepPINK: reproducible feature selection in deep neural networks Y ang Young Lu
Deep learning has become increasingly popular in both supervised and unsupervised machine learning thanks to its outstanding empirical performance.
Generalized Inverse Optimization through Online Learning
Chaosheng Dong, Yiran Chen, Bo Zeng
Neural Information Processing Systems http://nips.cc/
Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes
Loucas Pillaud-Vivien, Alessandro Rudi, Francis Bach
Neural Information Processing Systems http://nips.cc/
Adaptive Online Learning in Dynamic Environments
Lijun Zhang, Shiyin Lu, Zhi-Hua Zhou
In the literature, there are two different forms of dynamic regret.
Interactive Structure Learning with Structural Query-by-Committee
Christopher Tosh, Sanjoy Dasgupta
We present a generalization of the query-by-committee active learning algorithm for this setting, and we study its consistency and rate of convergence, both theoretically and empirically, with and without noise.