Hey guys, just thought some of you might enjoy this. I've set up an community AMI on aws that I've done a simul-build of a lot of the bleeding-edge releases of some of the most popular machine learning/deep learning packages/modules around.

If you're not using deep learning already, you should be. That was the message from legendary Google engineer Jeff Dean at the end of his keynote earlier this year at a conference on web search and data mining. Dean was referring to the rapid increase in machine learning algorithms' accuracy, driven by recent progress in deep learning, and the still untapped potential of these improved algorithms to change the world we live in and the products we build. But breakthroughs in deep learning aren't the only reason this is a big moment for machine learning. Just as important is that over the last five years, machine learning has become far more accessible to nonexperts, opening up access to a vast group of people. For most software developers, there have historically been many barriers to entry in machine learning, most notably software libraries designed more for academic researchers than for software engineers as well as a lack of sufficient data.

There are many deep learning resources freely available online, but it can be confusing knowing where to begin. Go from vague understanding of deep neural networks to knowledgeable practitioner in 7 steps! Deep learning is a branch of machine learning, employing numerous similar, yet distinct, deep neural network architectures to solve various problems in natural language processing, computer vision, and bioinformatics, among other fields. Deep learning has experienced a tremendous recent research resurgence, and has been shown to deliver state of the art results in numerous applications. In essence, deep learning is the implementation of neural networks with more than a single hidden layer of neurons.

Honda Motor Co. will spearhead its artificial intelligence efforts out of a new lab in Tokyo so that researchers can work closely with its engineers to commercialize the technology. Honda, based in Tokyo, will start the r&d center next year and combine existing AI teams in Silicon Valley, Europe and Japan at the downtown location, according to Yoshiyuki Matsumoto, president of the automaker's largely independent research arm. In choosing Tokyo over Silicon Valley, the carmaker is betting closer interaction between its scientists and developers will lead to AI-enabled products consumers want, he said in an interview. Advances in artificial intelligence are sprouting like "bamboo shoots after rain," so it's time to find commercial uses for the technology by marrying research with Japan's traditional strength in hardware, Matsumoto said. "We won't make much difference if we did the same things as everyone else in Silicon Valley. And not everyone has succeeded there."

In the wake of the massive DDoS attack that brought down huge swaths of the internet last week, strengthening cybersecurity is on everyone's minds. It's tough to devise systems that are truly secure, because as security evolves, so do the people trying to hack those systems. But a new experiment from Google showed a possible way forward using artificial intelligence. Google created an encryption game using three distinct entities--Alice, Eve, and Bob--created by deep learning neural networks. Alice sent thousands of 16 character strings of zeros and ones to Bob, encrypting them with a shared key. Eve sat in the middle, trying to decrypt the messages.

While the number of headlines about machine learning might lead one to think that we just discovered something profoundly new, the reality is that the technology is nearly as old as computing. It's no coincidence that Alan Turing, one of the most influential computer scientists of all time, started his 1950 treatise on computing with the question "Can machines think?" From our science fiction to our research labs, we have long questioned whether the creation of artificial versions of ourselves will somehow help us uncover the origin of our own consciousness, and more broadly, our role on earth. Unfortunately, the learning curve on AI is really damn steep. By tracing a bit of history, we should hopefully be able to get to the bottom of wtf machine learning really is.

Until recently, [Europe's] contribution to the innovation and commercialisation of machine intelligence technologies has been under-appreciated. We now see growing self-confidence borne of the success, and continued presence, of local acquired startups like VocalIQ, Swiftkey, Deepmind, Magic Pony Technology, and PredictionIO. London is Europe's startup centre, mixing capital, proximity to markets, and world-class research hubs.

We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The TS network copies its entire input vector to different locations in an expanded representation, with the location determined by its hidden unit activity. In this way, even a simple linear readout from the TS representation can implement a highly expressive deep-network-like function. The TS network hence avoids the vanishing gradient problem by construction, at the cost of larger representation size. We develop several methods to train the TS network, including equivalent kernels for infinitely wide and deep TS networks, a one-pass linear learning algorithm, and two backpropagation-inspired representation learning algorithms. Our experimental results demonstrate that the TS network is indeed more expressive and consistently learns faster than standard ReLU networks.

Learning with sequential data is widely encountered in domains such as natural language processing, genomics, speech recognition, video processing, financial time series analysis, and so on. Recurrent neural networks (RNN) are a flexible class of sequential models which can memorize past information, and selectively pass it on across sequence steps on multiple scales. However, training RNNs is challenging in practice, and backpropagation suffers from exploding and vanishing gradients as the length of the training sequence grows. To overcome this, either RNNs are trained over short sequences or incorporate more complex architectures such as long short-term memories (LSTM). For a detailed overview of RNNs, see [20]. Figure 1 contrasts the RNN with a feedforward neural network which has no memory. On the theoretical front, understanding of RNNs is at best rudimentary. With the current techniques, it is not tractable to analyze the highly nonlinear state evolution in RNNs. Analysis of backpropagation is also intractable due to non-convexity of the loss function, and in general, reaching the global optimum is hard. Here, we take the first steps towards addressing these challenging issues.

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second, we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.