We show that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all known activation functions, including rectified linear units, sigmoids and thresholds, and formally demonstrates that depth -- even if increased by 1 -- can be exponentially more valuable than width for standard feedforward neural networks. Moreover, compared to related results in the context of Boolean functions, our result requires fewer assumptions, and the proof techniques and construction are very different.

ABSTRACT Recurrent neural networks (RNNs), particularly long short-term memory (LSTM), have gained much attention in automatic speech recognition (ASR). Although some successful stories have been reported, training RNNs remains highly challenging, especially with limited training data. Recent research found that a well-trained model can be used as a teacher to train other child models, by using the predictions generated by the teacher model as supervision. This knowledge transfer learning has been employed to train simple neural nets with a complex one, so that the final performance can reach a level that is infeasible to obtain by regular training. In this paper, we employ the knowledge transfer learning approach to train RNNs (precisely LSTM) using a deep neural network (DNN) model as the teacher. This is different from most of the existing research on knowledge transfer learning, since the teacher (DNN) is assumed to be weaker than the child (RNN); however, our experiments on an ASR task showed that it works fairly well: without applying any tricks on the learning scheme, this approach can train RNNs successfully even with limited training data.

Trainees will learn essential techniques for real machine-learning model development, helping them to build more accurate models. In the masterclass, participants will work to deploy, test, and improve their models.

After an up-and-down start to the year, Chinese search giant Baidu issued earnings last week that outperformed on a host of key indicators. As we've come to expect from Baidu, revenue growth remained brisk, increasing at a healthy 31% year-over-year pace to total 2.5 billion. In keeping with its recent quarters, increased spending crimped Baidu's operating profits, which grew only 2.6% compared with the first quarter of 2015. Either way, Baidu's earnings exceeded expectations on the top and bottom line. What's more, Baidu's guidance for second-quarter sales proved better than analysts anticipated, sending the company's shares up in after-hours trading the day of the announcement.

After an up-and-down start to the year, Chinese search giant Baidu (NASDAQ:BIDU) issued earnings last week that outperformed on a host of key indicators. As we've come to expect from Baidu, revenue growth remained brisk, increasing at a healthy 31% year-over-year pace to total 2.5 billion. In keeping with its recent quarters, increased spending crimped Baidu's operating profits, which grew only 2.6% compared with the first quarter of 2015. Either way, Baidu's earnings exceeded expectations on the top and bottom line. What's more, Baidu's guidance for second-quarter sales proved better than analysts anticipated, sending the company's shares up in after-hours trading the day of the announcement.

The scikit-learn library is one of the most popular platforms for everyday machine learning and data science. The reason is because it is built upon Python, a fully featured programming language. But how do you get started with machine learning with scikit-learn. Kevin Markham is a data science trainer who created a series of 9 videos that show you exactly how to get started in machine learning with scikit-learn. In this post you will discover this series of videos and exactly what is covered, step-by-step to help you decide if the material will be useful to you.

Guest blog post by Xiu-Shen Wei, originally posted here. Deep Neural Networks, especially Convolutional Neural Networks (CNN), allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-arts in visual object recognition, object detection, text recognition and many other domains such as drug discovery and genomics. In addition, many solid papers have been published in this topic, and some high quality open source CNN software packages have been made available. There are also well-written CNN tutorials or CNN software manuals. However, it might lack a recent and comprehensive summary about the details of how to implement an excellent deep convolutional neural networks from scratch. Thus, we collected and concluded many implementation details for DCNNs. Here we will introduce these extensive implementation details, i.e., tricks or tips, for building and training your own deep networks. We assume you already know the basic knowledge of deep learning, and here we will present the implementation details (tricks or tips) in Deep Neural Networks, especially CNN for image-related tasks, mainly in eight aspects: 1) data augmentation; 2) pre-processing on images; 3) initializations of Networks; 4) some tips during training; 5) selections of activation functions; 6) diverse regularizations; 7)some insights found from figures and finally 8) methods of ensemble multiple deep networks. Additionally, the corresponding slides are available at [slide].

This is a self contained software accompanying the paper titled: Sequence Level Training with Recurrent Neural Networks: http://arxiv.org/abs/1511.06732. The code allows you to reproduce our result on the machine translation task. The code implements MIXER; it runs both training and evaluation. The software is written in Lua. Download the files in an appropriate directory and run the code from there.

One is going to give a historical overview about GoogleBrain and analyse the pattern, then we will conculde our finding about the current sitation and future changes in search engine algorithm. Back in 2006 there were some interests in implementing artificial intelligence in Google search engine algorithm. A few years later in 2014, GoogleBrain was established after acquisition of DeepMind, a British artificial intelligence company which was founded in 2010. They worked on how to play video games based on machine learning and artificial neural networks (ANNs). The smart artificial intelligence revolution can recognize patterns in digital representations of sounds, images and data.

The following interview is one of many included in the report. As part of my ongoing series of interviews surveying the frontiers of machine intelligence, I recently interviewed Yoshua Bengio. Bengio is a professor with the department of computer science and operations research at the University of Montreal, where he is head of the Machine Learning Laboratory (MILA) and serves as the Canada Research Chair in statistical learning algorithms. The goal of his research is to understand the principles of learning that yield intelligence. Yoshua Bengio: I have been researching neural networks since the '80s.