Going to school for a formal degree program for isn't always possible or desirable. For those considering an autodidactic alternative, this is for you. There's too much to learn, and the field is advancing rapidly. Master foundational concepts and then focus on projects in a specific domain of interest -- whether it's natural language understanding, computer vision, deep reinforcement learning, robotics, or whatever else. Motivation is far more important than micro-optimizing a learning strategy for some long-term academic or career goal.

In the blog "From Autonomous to Smart: Importance of Artificial Intelligence," we laid out the artificial intelligence (AI) challenges in creating "smart" edge devices: We also talked about how Moore's Law isn't going to bail us out of these challenges; that the growth of Internet of Things (IOT) data and the complexity of the problems that we are trying to address at the edge (think "smart" cars) is growing much faster than Moore's Law can accommodate. So we are going to use this blog to deep dive into the category of artificial intelligence called reinforcement learning. We are going to see how reinforcement learning might help us to address these challenges; to work smarter at the edge when brute force technology advances will not suffice. With the rapid increases in computing power, it's easy to get seduced into thinking that raw computing power can solve problems like smart edge devices (e.g., cars, trains, airplanes, wind turbines, jet engines, medical devices). Look at the dramatic increase in the number of possible moves between checkers and chess even though the board layout is exactly the same.

Machine Learning refers to a lot of things in a vast field which is rapidly growing. Arthur Samuel (1901-1990), an American pioneer in the field of computer gaming and artificial intelligence, coined the term "machine learning" in 1959. He defined it as a "field of study that gives computers the ability to learn without being explicitly programmed". Today, many say that machine learning is the future, although I'd say is already part of lots of things we have today. Machine learning, as a type of artificial intelligence (AI), enables computers to learn without being explicitly programmed, and to improve their functions when exposed to new data. By analyzing patterns in this data, the machine learning algorithms are self-adjusting based on a set of design rules.

In recent years, we've seen a resurgence in AI, or artificial intelligence, and machine learning. Machine learning has led to some amazing results, like being able to analyze medical images and predict diseases on-par with human experts. Google's AlphaGo program was able to beat a world champion in the strategy game go using deep reinforcement learning. Machine learning is even being used to program self driving cars, which is going to change the automotive industry forever. Imagine a world with drastically reduced car accidents, simply by removing the element of human error.

No real-world reward function is perfect. Sensory errors and software bugs may result in RL agents observing higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formalise this problem as a generalised Markov Decision Problem called Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards. Two ways around the problem are investigated. First, by giving the agent richer data, such as in inverse reinforcement learning and semi-supervised reinforcement learning, reward corruption stemming from systematic sensory errors may sometimes be completely managed. Second, by using randomisation to blunt the agent's optimisation, reward corruption can be partially managed under some assumptions.

The Hype Cycle for Emerging Technologies, 2017 provides insights gained from evaluations of more than 2,000 technologies the research and advisory firms tracks. From this large base of technologies, the technologies that show the most potential for delivering a competitive advantage over the next five to 10 years are included in the Hype Cycle. The eight technologies added to the Hype Cycle this year include 5G, Artificial General Intelligence, Deep Learning, Deep Reinforcement Learning, Digital Twin, Edge Computing, Serverless PaaS and Cognitive Computing. Ten technologies not included in the hype cycle for 2017 include 802.11ax, The three most dominant trends include Artifical Intelligence (AI) Everywhere, Transparently Immersive Experiences, and Digital Platforms.

As you may have gathered, the families of two-class classification, multi-class classification, anomaly detection, and regression are all closely related. Entirely different sets of data science questions belong in the extended algorithm families of unsupervised and reinforcement learning. Another family of unsupervised learning algorithms are called dimensionality reduction techniques. These are called reinforcement learning (RL) algorithms.

Reinforcement learning is concerned with identifying reward-maximizing behaviour policies in environments that are initially unknown. State-of-the-art reinforcement learning approaches, such as deep Q-networks, are model-free and learn to act effectively across a wide range of environments such as Atari games, but require huge amounts of data. Model-based techniques are more data-efficient, but need to acquire explicit knowledge about the environment. In this paper, we take a step towards using model-based techniques in environments with a high-dimensional visual state space by demonstrating that it is possible to learn system dynamics and the reward structure jointly. Our contribution is to extend a recently developed deep neural network for video frame prediction in Atari games to enable reward prediction as well. To this end, we phrase a joint optimization problem for minimizing both video frame and reward reconstruction loss, and adapt network parameters accordingly. Empirical evaluations on five Atari games demonstrate accurate cumulative reward prediction of up to 200 frames. We consider these results as opening up important directions for model-based reinforcement learning in complex, initially unknown environments.

When encountering novel objects, humans are able to infer a wide range of physical properties such as mass, friction and deformability by interacting with them in a goal driven way. This process of active interaction is in the same spirit as a scientist performing experiments to discover hidden facts. Recent advances in artificial intelligence have yielded machines that can achieve superhuman performance in Go, Atari, natural language processing, and complex control problems; however, it is not clear that these systems can rival the scientific intuition of even a young child. In this work we introduce a basic set of tasks that require agents to estimate properties such as mass and cohesion of objects in an interactive simulated environment where they can manipulate the objects and observe the consequences. We found that deep reinforcement learning methods can learn to perform the experiments necessary to discover such hidden properties. By systematically manipulating the problem difficulty and the cost incurred by the agent for performing experiments, we found that agents learn different strategies that balance the cost of gathering information against the cost of making mistakes in different situations. We also compare our learned experimentation policies to randomized baselines and show that the learned policies lead to better predictions.

The Hype Cycle for Emerging Technologies, 2017 provides insights gained from evaluations of more than 2,000 technologies the research and advisory firms tracks. From this large base of technologies, the technologies that show the most potential for delivering a competitive advantage over the next five to 10 years are included in the Hype Cycle. The eight technologies added to the Hype Cycle this year include 5G, Artificial General Intelligence, Deep Learning, Deep Reinforcement Learning, Digital Twin, Edge Computing, Serverless PaaS and Cognitive Computing. Ten technologies not included in the hype cycle for 2017 include 802.11ax, The three most dominant trends include Artifical Intelligence (AI) Everywhere, Transparently Immersive Experiences, and Digital Platforms.