At the heart of deep learning lies a hard optimization problem. So hard that for several decades after the introduction of neural networks, the difficulty of optimization on deep neural networks was a barrier to their mainstream usage and contributed to their decline in the 1990s and 2000s. Since then, we have overcome this issue. In this post, I explore the "hardness" in optimizing neural networks and see what the theory has to say. In a nutshell: the deeper the network becomes, the harder the optimization problem becomes.

It seems like we've perennially been on the edge of major breakthroughs in AI (artificial intelligence), virtual reality, personal robots, and other such cool tech for the past two decades. The first set of science fiction imaginings came true rather rapidly -- think trans-continental air travel, space stations, even drone warfare -- but it appears that the emergence of next-gen tech wizardry has stalled. But while we still can't chat with the on-board computer on our personal spaceship, artificial intelligence is far more pervasive in our daily lives today than most of us realize. As anyone who has trained their mobile phone assistant can attest, years of painstaking research and investment in artificial intelligence technologies is starting to yield impressive results. Siri can predict our commute patterns, Microsoft Cortana warns us of bad weather, and the Google Assistant diligently sets calendar reminders with the impassive demeanor of an English butler of yore.

What is A.I. or artificial intelligence but the'space of possible minds', argues Murray Shanahan, scientific advisor on the movie Ex Machina and Professor of Cognitive Robotics at Imperial College London. But where are we now in the A.I. evolution? What players do we think will lead, if not win, the current race? And how should we think about issues such as ethics and automation of jobs without descending into obvious extremes? All this and more, including a surprise easter egg in Ex Machina shared by Shanahan, whose work influenced the movie.

A year ago, I wrote that I had seen the future. I had attended the IBM Insights conference in Las Vegas, an entire conference devoted to the Internet of Things, analytics, cognitive marketing, and related topics. And if the concept of Moore's Law (that technology doubles every year, or every 18 months or two years, depending on what you read) holds true, then a lot has happened since then. While last year's future is now a history lesson, what is currently happening in the World of Watson is moving at warp speed! This past week more than 17,000 people attended IBM's World of Watson, the new name of the conference devoted to cognitive technology. Fifteen "influencers" were invited to attend IBM's first-ever Cognitive College that included product demonstrations and lectures about how Watson's cognitive abilities are impacting marketing and commerce.

Pixar founder George Lucas may have given the world the word "Droid" before he sold the company to Apple's Steve Jobs, but Apple CEO Tim Cook makes no secret that he wants his company to play its part in making intelligence in machines a reality. Cook talked up Apple's efforts in augmented reality just last week. This week he's spilling at least some of the beans on his company's plans for Artificial Intelligence, though his vision seems more connected to machine intelligence and pattern recognition than the evolution of a smart bot like C3PO, at least right now. "A.I. is horizontal in nature, running across all products," he told Nikkei. He said it is used "in ways that most people don't even think about."

Chris Eliasmith, the director of the University of Waterloo's Centre for Theoretical Neuroscience, Canada, is trying to build a brain. Eliasmith's artificial model, Spaun, currently has just 2.5 million neurons to the human brain's 100 billion. But unlike more computationally demanding simulations, which have run for only a few simulated seconds, it's actually capable of doing something with them. "There's been an attitude of scale for scale's sake," Eliasmith explains. "But for us, the focus was discovering how neurones can be organised to produce behaviours, such as solving simple intelligence tasks."

Facebook could one day build facial gesture controls for its app thanks to the acquisition of a Carnegie Mellon University spinoff company called FacioMetrics. The startup made an app called Intraface that could detect seven different emotions in people's faces, but it's been removed from the app stores. The acquisition aligns with a surprising nugget of information Facebook slipped into a 32-bullet point briefing sent to TechCrunch this month. "Future applications of deep learning platform on mobile: Gesture-based controls, recognize facial expressions and perform related actions" It's not hard to imagine Facebook one day employing FacioMetrics' tech and its own AI to let you add a Like or one of its Wow/Haha/Angry/Sad emoji reactions by showing that emotion with your face. "How people share and communicate is changing and things like masks and other effects allow people to express themselves in fun and creative ways. We're excited to welcome the Faciometrics team who will help bring more fun effects to photos and videos and build even more engaging sharing experiences on Facebook."

In business you have to look at digital usage measurements as chess pieces--what pieces are on the chessboard, how they move, and what movements they cause downstream, can impact future decisions. That downstream view usually includes A/B and multivariate testing. Optimization is typically deployed to improve website elements that impact user experience. Changes in the optimization test platforms for Adobe and Google Analytics reflect the effect of machine learning on increasing the accuracy of test results and helping wmake better decisions. Adobe announced Auto-Target, a machine learning protocol that automates personalization-focused testing of elements to determine the preferred individual experience with media.

Raise your hand if you're a bit too optimistic when it comes to planning your day with a to-do list. If you're like me, chances are you write down too many tasks and always end up with unfinished tasks at the end of the day. Popular task-management service Todoist wants to help you reschedule your tasks and even out the work load using machine learning. Smart scheduling is a work in progress and will get better over time, but it sounds like a promising feature for intense Todoist users. "We created a couple of neural networks that can help our users with scheduling and rescheduling stuff. We think it's a pretty cool usage of AI," Todoist founder and CEO Amir Salihefendic told me.

There are approximately 1,100 known antimicrobial peptides (AMP) with diverse sequences that can permeate microbial membranes. To help discover the "blueprint" for natural AMP sequences, researchers from the University of Illinois at Urbana-Champaign and the University of California, Los Angeles, have developed a new machine learning approach to discover and design alpha-helical membrane active peptides based on their physicochemical properties. "In this work, we have trained a machine learning classifier--known as a support vector machine--to recognize membrane activity and experimentally calibrated the recognition metric by peptide synthesis and characterization," explained Andrew Ferguson, an assistant professor of materials science and engineering at Illinois. "We use machine learning to not only discover new membrane active peptides, but to also identify membrane activity in known peptides with previously defined functions leading us to discover membrane activity in diverse and unexpected peptide families. "Since getting cargo into a cell is important for many applications, we anticipate that this tool can have broad biomedical implications including in immunotherapy and in broad-spectrum membrane-active antimicrobial peptides to combat the rising incidence of drug resistance, design of cationic cell-penetrating peptides for nucleic acid transfection into cells, and in targeting and permeating anticancer therapeutics into tumors," added Ferguson, who was the senior computational investigator for the project. In this collaborative work, the Illinois researchers developed the computational innovations, with the experimental testing of the predictions accomplished at UCLA. The results, which highlight the difference between the efficacy of an antimicrobial and its recognizability as such, are surprising. "AMPs do not share a common core structure, but tend to be short, cationic, and amphiphilic," Ferguson said. "By training our machine learning classifier over a training set comprising peptides with known antimicrobial activity (hits) and decoy peptides with no activity (misses), the classifier learned the physical and chemical properties of a peptide that make for good membrane activity.