We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for convex optimization given their edge over stochastic gradient descent (SGD); but their theoretical analysis almost exclusively assumes convexity. In contrast, we prove non-asymptotic rates of convergence (to stationary points) of SVRG for nonconvex optimization, and show that it is provably faster than SGD and gradient descent. We also analyze a subclass of nonconvex problems on which SVRG attains linear convergence to the global optimum. We extend our analysis to mini-batch variants of SVRG, showing (theoretical) linear speedup due to mini-batching in parallel settings.

Automatic Offline Handwritten Signature Verification has been researched over the last few decades from several perspectives, using insights from graphology, computer vision, signal processing, among others. In spite of the advancements on the field, building classifiers that can separate between genuine signatures and skilled forgeries (forgeries made targeting a particular signature) is still hard. We propose approaching the problem from a feature learning perspective. Our hypothesis is that, in the absence of a good model of the data generation process, it is better to learn the features from data, instead of using hand-crafted features that have no resemblance to the signature generation process. To this end, we use Deep Convolutional Neural Networks to learn features in a writer-independent format, and use this model to obtain a feature representation on another set of users, where we train writer-dependent classifiers. We tested our method in two datasets: GPDS-960 and Brazilian PUC-PR. Our experimental results show that the features learned in a subset of the users are discriminative for the other users, including across different datasets, reaching close to the state-of-the-art in the GPDS dataset, and improving the state-of-the-art in the Brazilian PUC-PR dataset.

In the late 00's some clever academics rebranded a subset of neural network techniques to'Deep Learning', which just means a stack of different nets on top of one another, forming a sort of computationally-brilliant lasagne. When I say'machine learning' in this blogpost, I'm referring to some kind of neural network technique.) Robotics has just started to get into neural networks. This has already sped up development. This year, Google demonstrated a system that teaches robotic arms to learn how to pick up objects of any size and shape.

Change of Plans.... We are going to bump this months OC Deep Learning Meetup to Wed. the 13th as several people requested it and no one objected. Plus this gives us the pleasure of hearing from Manuel Beltran Chief Software Architect, Battle Command Systems at Boeing. Mr. Beltran has over 20 years computer engineering experience and has worked as a Knowledge Engineer on various NASA projects, his Expert Systems and Neural Networks have provided many years of safe and successful Space Shuttle launch pad operations. Mr. Beltran also provided Software Engineering leadership, vision, and technical innovation to the US Army's Brigade Combat Team Modernization, formerly known as the Future Combat System. As the Chief Software Architect of the Battle Command System, he has been responsible for the software development of the largest military modernization effort in the history of the US military.

Google has made one its largest European acquisitions to date with a deal to buy DeepMind technologies, a London-based artificial intelligence firm which specialises in machine learning, advanced algorithms and systems neuroscience. The Guardian understands that Google paid 400m ( 650m) for DeepMind, which develops technologies for e-commerce and games, and has demonstrated computer systems capable of playing computer games. It aims, it says, to develop computers that think like humans. The two-year-old artificial intelligence startup was founded by former child chess prodigy and neuroscientist Demis Hassabis alongside Shane Legg and Mustafa Suleyman. DeepMind has reportedly competed with Google and other major artificial intelligence companies for talent and Google's chief executive Larry Page is said to have led the deal himself.

The problem with most SP500 companies is that they tend to hide their analytic procedures, regardless of the specifics. They consider these procedures their trade secrets, especially as the procedures are usually tightly engineered around data they have, again considered trade secret. You won't hear them claiming they do deep learning, just like you won't hear them claiming they do linear regression. Your best bet is looking at job offers, checking what requirements they put towards new hires… or certain industry-specific conferences.

This is part 3/3 of a series on deep belief networks. Part 1 focused on the building blocks of deep neural nets – logistic regression and gradient descent. Part 2 focused on how to use logistic regression as a building block to create neural networks, and how to train them. Part 3 will focus on answering the question: "What is a deep belief network?" and the algorithms we use to do training and prediction. In its simplest form, a deep belief network looks exactly like the artificial neural networks we learned about in part 2! As long as there is at least 1 hidden layer, the model is considered to be "deep".

Google confirmed they plan to improve Google Translate's accuracy through artificial intelligence called deep learning. Deep learning is an advanced model of machine learning where an algorithm takes what it has already "learned" (data previously processed) and uses it to form new ways to solve problems in a pattern. Jeff Dean, Google Senior Fellow, confirmed that his team has been working with the Google Translate team to "scale out experiments with translation based on deep learning." Deep learning and technologies derived from it, including deep and recurrent neural nets, are objectively excellent at tackling sequential problems such as speech and image recognition, as long as there is sufficient existing material to train them. On that front, Google Translate's vast data trove of translated material should indeed prove quite useful.

Mastering arcade games seems cute by comparison. Researchers at DeepMind, the Google-owned artificial intelligence lab, announced Wednesday they had achieved a breakthrough not thought possible for at least another decade: a computer program that defeats humans at Go, an enormously complicated strategy game. See Also: This robot can solve Rubik's Cube in one second This network was named by EContent Magazine to its "Trendsetting Products of 2014" list.

Yann LeCun invited me to speak at this week's inaugural symposium on "The Future of AI" at NYU. It's an amazing gathering of leaders in the field to discuss the state of AI and its continued advancement. Here's what I talked about -- how deep learning is a new software model that needs a new computing model; why AI researchers have adopted GPU-accelerated computing; and NVIDIA's ongoing efforts to advance AI as we enter into its exponential adoption. And why, after all these years, AI has taken off. For as long as we have been designing computers, AI has been the final frontier.