Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effect on the underlying loss landscape, is not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comp arisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network quantization (SLQ) for high-bit quantization and multi-level network quantization (MLQ) for extremely low-bit quantization (ternary).We are the first to consider the network quantization from both width and depth level. In the width level, parameters are divided into two parts: one for quantization and the other for re-training to eliminate the quantization loss. SLQ leverages the distribution of the parameters to improve the width level. In the depth level, we introduce incremental layer compensation to quantize layers iteratively which decreases the quantization loss in each iteration. The proposed approaches are validated with extensive experiments based on the state-of-the-art neural networks including AlexNet, VGG-16, GoogleNet and ResNet-18. Both SLQ and MLQ achieve impressive results.

It asks to use an LSTM and predict the parity of bit sequences of length 50. Naive attempts didn't work (large hidden state, different rnn cells, different optimization algos, etc). It was a fun challenge. I like that it's very easy to state but requires a bit of insight to figure out. Do you know of any others like that?

Have you noticed that interest in artificial intelligence (AI) has really taken off in the last year or so? A lot of that interest is fueled by deep learning. Deep learning has revolutionized the way we use our phones, bringing us new applications such as Google Voice and Apple's Siri, which are based on AI models trained using deep learning. Deep learning is a new machine learning method based on neural networks that learns and becomes more accurate as we feed the model more data. A widely-accepted principle of deep learning is shown on the left-hand side of the chart below: deep learning–based AI models have much higher accuracy than traditional machine learning methods, but require much more data to train to achieve that accuracy.

I would recommend Vision by David Marr (1982). It studies early vision relating neurophysiology to algorithms and assumptions about the physical world. The key point it's making is there are three different levels of explanation when studying how the brain achieve s a certain task, and one should try to keep them separate if one wants to achieve clear understanding. The sort of thing he is criticising is fuzzy thinking like "deep learning is inspired by how the brain works". So for stereopsis he goes from psychological experiments looking at visual illusions and random dot stereograms ( magic eye?) which illustrate the assumptions about the 3d world that allow us to recover depth from 2 2d images, to possible neurophysiological implementation s.

China's leading technology companies are on fire, heavily investing in artificial intelligence and building true global presences. McKinsey recently reported that academic and research institutions in the country publish more cited research papers than the US, UK, or any other global leader in AI, producing nearly 10,000 papers in 2015 alone. Backed by strong government mandates and billions of dollars of both private and public investments, China is challenging the US for position of global AI leader. Fearful of competition, the US government is considering placing restrictions on Chinese investments in AI and technology in the United States. In many sectors, such as healthcare, China may already be ahead of America in applying AI to critical public issues.

Role of Artificial Intelligence in FinTech is getting more and more importance. AI is actually innovating the whole of traditional FinTech. We will discuss in detail about AI role in FinTech's discoveries, inventions and innovations. It was big and very important event for the FinTech and Banking industry in Khartoum Sudan. The opening speech was done by Sudan central bank officials which was the main attraction.

Computers may better predict taxi and ride sharing service demand, paving the way toward smarter, safer and more sustainable cities, according to an international team of researchers. In a study, the researchers used two types of neural networks -- computational systems modeled on the human brain -- that analyzed patterns of taxi demand. This deep learning approach, which lets computers learn on their own, was then able to predict the demand patterns significantly better than current technology. "Ride sharing companies, like Uber in the United States, and Didi Chuxing in China, are becoming more and more popular and have really changed the way people approach transportation," said Jessie Li, associate professor of information sciences and technology, Penn State. "And you can imagine how important it would be to predict the taxi demand because the taxi company could dispatch the cars even before the need arises."

We introduce Dynamic Deep Neural Networks (D2NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D2NN neurons are executed, and the particular subset is determined by the D2NN itself. By pruning unnecessary computation depending on input, D2NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D2NN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D2NN is trained end to end. Both regular and controller modules in a D2NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D2NN architectures on image classification tasks, we demonstrate that D2NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.

We introduce Recurrent Predictive State Policy (RPSP) networks, a recurrent architecture that brings insights from predictive state representations to reinforcement learning in partially observable environments. Predictive state policy networks consist of a recursive filter, which keeps track of a belief about the state of the environment, and a reactive policy that directly maps beliefs to actions, to maximize the cumulative reward. The recursive filter leverages predictive state representations (PSRs) (Rosencrantz and Gordon, 2004; Sun et al., 2016) by modeling predictive state-- a prediction of the distribution of future observations conditioned on history and future actions. This representation gives rise to a rich class of statistically consistent algorithms (Hefny et al., 2018) to initialize the recursive filter. Predictive state serves as an equivalent representation of a belief state. Therefore, the policy component of the RPSP-network can be purely reactive, simplifying training while still allowing optimal behaviour. Moreover, we use the PSR interpretation during training as well, by incorporating prediction error in the loss function. The entire network (recursive filter and reactive policy) is still differentiable and can be trained using gradient based methods. We optimize our policy using a combination of policy gradient based on rewards (Williams, 1992) and gradient descent based on prediction error. We show the efficacy of RPSP-networks under partial observability on a set of robotic control tasks from OpenAI Gym. We empirically show that RPSP-networks perform well compared with memory-preserving networks such as GRUs, as well as finite memory models, being the overall best performing method.