AI Researcher, Cognitive Technologist Inventor - AI Thinking, Think Chain Innovator - AIOT, XAI, Autonomous Cars, IIOT Founder Fisheyebox Spatial Computing Savant, Transformative Leader, Industry X.0 Practitioner How much might cost Real AI Model? Encyclopedic Intelligent Systems has developed the first real model of the Real/Causal AI, including the following elements of its Universal Intelligent Platform, I-World: Machine World Model; Master Algorithm, Causal.World; World Data Framework, World.Data; Global Knowledge Base, World.Net; Domain Knowledge Base, Domain.Net; The Development has reached a stage of a proof of principle, concept and mechanism in which the best AI technology stack of hardware, software and dataware is constructed and tested to explore and demonstrate the feasibility of the Real/Causal AI Model. It creates a world-data mapping of all possible entities, their relationships and behaviors, binding causes and effects. To build truly AI machines of infinitely powerful digital intelligence, we need to encode, program or teach them what the world is with all its complex cause-effect relationships. What makes machine intelligence and learning a true and real AI is the powerful underlying causal master algorithms used to reveal the causal patterns in the world's data universe.

Continuous first-person 3D environments pose unique exploration challenges to reinforcement learning (RL) agents because of their high-dimensional state and action spaces. These challenges can be ameliorated by using semantically meaningful state abstractions to define novelty for exploration. We propose that learned representations shaped by natural language provide exactly this form of abstraction. In particular, we show that vision-language representations, when pretrained on image captioning datasets sampled from the internet, can drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by comparing the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains -- one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world -- as well as two popular deep RL algorithms: Impala and R2D2. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.

We show that a purely tactile dextrous in-hand manipulation task with continuous regrasping, requiring permanent force closure, can be learned from scratch and executed robustly on a torque-controlled humanoid robotic hand. The task is rotating a cube without dropping it, but in contrast to OpenAI's seminal cube manipulation task, the palm faces downwards and no cameras but only the hand's position and torque sensing are used. Although the task seems simple, it combines for the first time all the challenges in execution as well as learning that are important for using in-hand manipulation in real-world applications. We efficiently train in a precisely modeled and identified rigid body simulation with off-policy deep reinforcement learning, significantly sped up by a domain adapted curriculum, leading to a moderate 600 CPU hours of training time. The resulting policy is robustly transferred to the real humanoid DLR Hand-II, e.g., reaching more than 46 full 2${\pi}$ rotations of the cube in a single run and allowing for disturbances like different cube sizes, hand orientation, or pulling a finger.

Today we're joined by Doina Precup, a research team lead at DeepMind Montreal, and a professor at McGill University. In our conversation with Doina, we discuss her recent research interests, including her work in hierarchical reinforcement learning, with the goal being agents learning abstract representations, especially over time. We also explore her work on reward specification for RL agents, where she hypothesizes that a reward signal in a complex environment could lead an agent to develop attributes of intuitive intelligence. We also dig into quite a few of her papers, including On the Expressivity of Markov Reward, which won a NeruIPS 2021 outstanding paper award. Finally, we discuss the analogy between hierarchical RL and CNNs, her work in continual RL, and her thoughts on the evolution of RL in the recent past and present, and the biggest challenges facing the field going forward.

Abstract: Digital twin (DT) technologies have emerged as a solution for real-time data-driven modeling of cyber physical systems (CPS) using the vast amount of data available by Internet of Things (IoT) networks. In this position paper, we elucidate unique characteristics and capabilities of a DT framework that enables realization of such promises as online learning of a physical environment, real-time monitoring of assets, Monte Carlo heuristic search for predictive prevention, on-policy, and off-policy reinforcement learning in real-time. We establish a conceptual layered architecture for a DT framework with decentralized implementation on cloud computing and enabled by artificial intelligence (AI) services for modeling, event detection, and decision-making processes. The DT framework separates the control functions, deployed as a system of logically centralized process, from the physical devices under control, much like software-defined networking (SDN) in fifth generation (5G) wireless networks. We discuss the moment of the DT framework in facilitating implementation of network-based control processes and its implications for critical infrastructure. To clarify the significance of DT in lowering the risk of development and deployment of innovative technologies on existing system, we discuss the application of implementing zero trust architecture (ZTA) as a necessary security framework in future data-driven communication networks.

When I first studied policy gradient algorithms, I did not find them particularly easy to fathom. Intuitively they seemed straightforward enough -- sample actions, observe rewards, tweak the policy -- but after the initial idea followed many lengthy derivations, calculus tricks I had long forgotten, and an overwhelming amount of notation. At a certain point, it just became a blur of probability distributions and gradients. In this article, I try to explain the concept step by step, including key thought processes and mathematical operations. Admittedly, it's a bit of a long read and requires a certain preliminary knowledge on Reinforcement Learning (RL), but hopefully it sheds some light on the idea behind policy gradients. The focus is on likelihood ratio policy gradients, which is the foundation of classical algorithms such as REINFORCE/vanilla policy gradient.

Ever since I got curious and hooked on to the field of reinforcement learning and its numerous applications for the industry, my excitement for the field has only gotten stronger by the day. Here, I'd like to share some of my learnings about the potential applications of reinforcement learning (RL) for consumer businesses. But, before I dive into the details, a quick introduction about RL for ML practitioners who are new to the subject. RL is a branch of Machine Learning involving the training of smart agent that can learn to perform a goal through trial & error in an environment and at the end of the training we have an agent that can perform the goal in real life independently. Now, if you are familiar with the other types of ML -- supervised & unsupervised learning techniques--this might sound very similar to the supervised learning approach.

Predicting how climate and the environment will change over time or how air flows over an aircraft are problems too complex even for the most powerful supercomputers to solve. Scientists rely on models to fill in the gap between what they can simulate and what they need to predict. But, as every meteorologist knows, models often rely on partial or even faulty information which may lead to bad predictions. Now, researchers from the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) are forming what they call "intelligent alloys", combining the power of computational science with artificial intelligence to develop models that complement simulations to predict the evolution of science's most complex systems. In a paper published in Nature Communications, Petros Koumoutsakos, the Herbert S. Winokur, Jr. Professor of Engineering and Applied Sciences and co-author Jane Bae, a former postdoctoral fellow at the Institute of Applied Computational Science at SEAS, combined reinforcement learning with numerical methods to compute turbulent flows, one of the most complex processes in engineering.

Researchers have combined reinforcement learning with numerical methods to compute turbulent flows, one of the most complex processes in …