In this paper, we leverage ideas from model-based control to address the sample efficiency problem of reinforcement learning (RL) algorithms. Accelerating learning is an active field of RL highly relevant in the context of time-varying systems. Traditional transfer learning methods propose to use prior knowledge of the system behavior to devise a gradual or immediate data-driven transformation of the control policy obtained through RL. Such transformation is usually computed by estimating the performance of previous control policies based on measurements recently collected from the system. However, such retrospective measures have debatable utility with no guarantees of positive transfer in most cases. Instead, we propose a model-based transformation, such that when actions from a control policy are applied to the target system, a positive transfer is achieved. The transformation can be used as an initialization for the reinforcement learning process to converge to a new optimum. We validate the performance of our approach through four benchmark examples. We demonstrate that our approach is more sample-efficient than fine-tuning with reinforcement learning alone and achieves comparable performance to linear-quadratic-regulators and model-predictive control when an accurate linear model is known in the three cases. If an accurate model is not known, we empirically show that the proposed approach still guarantees positive transfer with jump-start improvement.

Faults are endemic to all systems. Adaptive fault-tolerant control maintains degraded performance when faults occur as opposed to unsafe conditions or catastrophic events. In systems with abrupt faults and strict time constraints, it is imperative for control to adapt quickly to system changes to maintain system operations. We present a meta-reinforcement learning approach that quickly adapts its control policy to changing conditions. The approach builds upon model-agnostic meta learning (MAML). The controller maintains a complement of prior policies learned under system faults. This "library" is evaluated on a system after a new fault to initialize the new policy. This contrasts with MAML, where the controller derives intermediate policies anew, sampled from a distribution of similar systems, to initialize a new policy. Our approach improves sample efficiency of the reinforcement learning process. We evaluate our approach on an aircraft fuel transfer system under abrupt faults.

We propose a novel adaptive reinforcement learning control approach for fault tolerant control of degrading systems that is not preceded by a fault detection and diagnosis step. Therefore, \textit{a priori} knowledge of faults that may occur in the system is not required. The adaptive scheme combines online and offline learning of the on-policy control method to improve exploration and sample efficiency, while guaranteeing stable learning. The offline learning phase is performed using a data-driven model of the system, which is frequently updated to track the system's operating conditions. We conduct experiments on an aircraft fuel transfer system to demonstrate the effectiveness of our approach.

A desirable property in fault-tolerant controllers is adaptability to system changes as they evolve during systems operations. An adaptive controller does not require optimal control policies to be enumerated for possible faults. Instead it can approximate one in real-time. We present two adaptive fault-tolerant control schemes for a discrete time system based on hierarchical reinforcement learning. We compare their performance against a model predictive controller in presence of sensor noise and persistent faults. The controllers are tested on a fuel tank model of a C-130 plane. Our experiments demonstrate that reinforcement learning-based controllers perform more robustly than model predictive controllers under faults, partially observable system models, and varying sensor noise levels.

Computational Thinking (CT) is considered a core competency in problem formulation and problem solving. We have developed the Computational Thinking using Simulation and Modeling (CTSiM) learning environment to help middle school students learn science and CT concepts simultaneously. In this paper, we present an approach that leverages multiple linked representations to help students learn by constructing and analyzing computational models of science topics. Results from a recent study show that students successfully use the linked representations to become better modelers and learners.

This paper discusses an expert system shell that integrates rule-based reasoning and the Dempster-Shafer evidence combination scheme. Domain knowledge is stored as rules with associated belief functions. The reasoning component uses a combination of forward and backward inferencing mechanisms to allow interaction with users in a mixed-initiative format.

The Betty’s Brain computer-based learning system provides an open-ended and choice-rich environment for science learning. Using the learning-by-teaching paradigm paired with feedback and support provided by two pedagogical agents, the system also promotes the development of self-regulated learning strategies to support preparation for future learning. We apply metacognitive learning theories and experiential analysis to interpret the results from previous classroom studies. We propose an integrated cognitive and metacognitive model for effective, self-regulated student learning in the Betty’s Brain environment, and then apply this model to interpret and analyze common suboptimal learning strategies students apply during their learning. This comparison is used to derive feedback for helping learners overcome these difficulties and adopt more effective strategies for regulating their learning. Preliminary results demonstrate that students who were responsive to the feedback had better learning performance.

The Association for the Advancement of Artificial Intelligence was pleased to present the 2010 Fall Symposium Series, held Thursday through Saturday, November 11-13, at the Westin Arlington Gateway in Arlington, Virginia. The titles of the eight symposia are as follows: (1) Cognitive and Metacognitive Educational Systems; (2) Commonsense Knowledge; (3) Complex Adaptive Systems: Resilience, Robustness, and Evolvability; (4) Computational Models of Narrative; (5) Dialog with Robots; (6) Manifold Learning and Its Applications; (7) Proactive Assistant Agents; and (8) Quantum Informatics for Cognitive, Social, and Semantic Processes. The highlights of each symposium are presented in this report.

The Association for the Advancement of Artificial Intelligence was pleased to present the 2010 Fall Symposium Series, held Thursday through Saturday, November 11-13, at the Westin Arlington Gateway in Arlington, Virginia. The titles of the eight symposia are as follows: (1) Cognitive and Metacognitive Educational Systems; (2) Commonsense Knowledge; (3) Complex Adaptive Systems: Resilience, Robustness, and Evolvability; (4) Computational Models of Narrative; (5) Dialog with Robots; (6) Manifold Learning and Its Applications; (7) Proactive Assistant Agents ; and (8) Quantum Informatics for Cognitive, Social, and Semantic Processes. The highlights of each symposium are presented in this report.

Our learning by teaching environment has students take on the role and responsibilities of a teacher to a virtual student named Betty. The environment is structured so that successfully instructing their teachable agent requires the students to learn and understand science topics for themselves. This process is supported by adaptive scaffolding and feedback from the system. This feedback is instantiated through the interactions with the teachable agent and a mentor agent, named Mr. Davis. This paper provides an overview of two studies that were conducted with 5th grade science students and a description of the analysis techniques that we have developed for interpreting students’ activities in this learning environment.