If you're not interested in how Recurrent Neural Networks work, feel free to jump down to the implementation. A Recurrent Neural Network (RNN) is a type of neural network that specializes in processing sequences. Given a seed "She walked her ___", an RNN might predict "dog". The trick with RNNs in text generation is using predictions as seeds for further predictions. One issue with standard neural networks as it applies to text geneation is that it has a fixed input and output size.

I'm going to do a comparison on recent (or at least lesser-known) gradient optimization methods. The ones I've encountered up to now are the following: I would be particularly interested in approaches not using any hyperparameters at all (such as number 3 - COCOB), however I will consider all of the interesting and promising methods. Are you aware of some novelties or lesser-known approaches? Previously I've posted the question in r/MLQuestions but haven't received any feedback so I'm posting it here, hope it's not violating any rules.

Learn to build cybertrolls detection engine with CNN, keras, Glove and popular programming language Python.NEW by Evergreen Technologies What you'll learn Detect cybertroll in social messages using CNN, Glove embeddings and Keras Description Course Description Learn to build cybertrolls detection engine with CNN, keras, Glove and popular programming language Python. Understanding of cybertrolls classification Understand the world of world embeddings Learn CNN from scratch Leverage CNN, Keras, Glove to classify cybertrolls in social messages Learn how to represent text as numeric vectors using glove embeddings Learn how to evaluate model from scratch User Jupyter Notebook for programming Build a real life web application to classify social messages A Powerful Skill at Your Fingertips Learning the fundamentals of text classification puts a powerful and very useful tool at your fingertips. Python and Jupyter are free, easy to learn, has excellent documentation. No prior knowledge of deep learning or Machine learning is assumed. I a, covering topics like CNN, Word Embeddings Precision, Recall in depth so that even beginners can understand this course very well.

Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation. Papers published at the Neural Information Processing Systems Conference.

Deep generative models (DGMs) are effective on learning multilayered representations of complex data and performing inference of input data by exploring the generative ability. However, little work has been done on examining or empowering the discriminative ability of DGMs on making accurate predictions. This paper presents max-margin deep generative models (mmDGMs), which explore the strongly discriminative principle of max-margin learning to improve the discriminative power of DGMs, while retaining the generative capability. We develop an efficient doubly stochastic subgradient algorithm for the piecewise linear objective. Empirical results on MNIST and SVHN datasets demonstrate that (1) max-margin learning can significantly improve the prediction performance of DGMs and meanwhile retain the generative ability; and (2) mmDGMs are competitive to the state-of-the-art fully discriminative networks by employing deep convolutional neural networks (CNNs) as both recognition and generative models. Papers published at the Neural Information Processing Systems Conference.

Numerous neurophysiological studies have revealed that a large number of the primary visual cortex neurons operate in a regime called surround modulation. Surround modulation has a substantial effect on various perceptual tasks, and it also plays a crucial role in the efficient neural coding of the visual cortex. Inspired by the notion of surround modulation, we designed new excitatory-inhibitory connections between a unit and its surrounding units in the convolutional neural network (CNN) to achieve a more biologically plausible network. Our experiments show that this simple mechanism can considerably improve both the performance and training speed of traditional CNNs in visual tasks. We further explore additional outcomes of the proposed structure.

A proper understanding of the striking generalization abilities of deep neural networks presents an enduring puzzle. Recently, there has been a growing body of numerically-grounded theoretical work that has contributed important insights to the theory of learning in deep neural nets. There has also been a recent interest in extending these analyses to understanding how multitask learning can further improve the generalization capacity of deep neural nets. These studies deal almost exclusively with regression tasks which are amenable to existing analytical techniques. We develop an analytic theory of the nonlinear dynamics of generalization of deep neural networks trained to solve classification tasks using softmax outputs and cross-entropy loss, addressing both single task and multitask settings.

Engineered proteins offer the potential to solve many problems in biomedicine, energy, and materials science, but creating designs that succeed is difficult in practice. A significant aspect of this challenge is the complex coupling between protein sequence and 3D structure, with the task of finding a viable design often referred to as the inverse protein folding problem. We develop relational language models for protein sequences that directly condition on a graph specification of the target structure. Our approach efficiently captures the complex dependencies in proteins by focusing on those that are long-range in sequence but local in 3D space. Our framework significantly improves in both speed and robustness over conventional and deep-learning-based methods for structure-based protein sequence design, and takes a step toward rapid and targeted biomolecular design with the aid of deep generative models.

Recurrent neural networks (RNNs) are a widely used tool for modeling sequential data, yet they are often treated as inscrutable black boxes. Given a trained recurrent network, we would like to reverse engineer it--to obtain a quantitative, interpretable description of how it solves a particular task. Even for simple tasks, a detailed understanding of how recurrent networks work, or a prescription for how to develop such an understanding, remains elusive. In this work, we use tools from dynamical systems analysis to reverse engineer recurrent networks trained to perform sentiment classification, a foundational natural language processing task. Given a trained network, we find fixed points of the recurrent dynamics and linearize the nonlinear system around these fixed points.

Combinatorial optimization problems are typically tackled by the branch-and-bound paradigm. We propose a new graph convolutional neural network model for learning branch-and-bound variable selection policies, which leverages the natural variable-constraint bipartite graph representation of mixed-integer linear programs. We train our model via imitation learning from the strong branching expert rule, and demonstrate on a series of hard problems that our approach produces policies that improve upon state-of-the-art machine-learning methods for branching and generalize to instances significantly larger than seen during training. Moreover, we improve for the first time over expert-designed branching rules implemented in a state-of-the-art solver on large problems. Papers published at the Neural Information Processing Systems Conference.