I present skill bootstrapping, a proposed new research direction for agent learning and planning that allows an agent to start with low-level primitive actions, and develop skills that can be used for higher-level planning. Skills are developed over the course of solving many different problems in a domain, using reinforcement learning techniques to complement the benefits and disadvantages of heuristic-search planning. I describe the overall architecture of the proposed approach and discuss how it relates to other work.

Reinforcement Learning is a theoretical framework for optimizing the behavior of artificial agents. The notion that behavior in the natural world is in some sense optimal is explored by areas such as biomechanics and physical anthropology. These fields propose a variety of candidate optimality criteria as possible formulations of the principles underlying natural motion. Recent developments in computational biomechanics allow us to create articulated models of living creatures with a significant degree of biological realism. I aim to bring these elements together in my research by using Reinforcement Learning to generate optimized behavior in biomechanical simulations. Such a generative approach will allow us to examine critically postulated optimality criteria and investigate hypotheses that cannot be easily studied in the real world.

The objective of my doctoral research is bring together two fields: partially-observable reinforcement learning (PORL) and non-parametric Bayesian statistics (NPB) to address issues of statistical modeling and decision-making in complex, real-world domains.

Proposed Thesis: Achievable set methods can efficiently compute useful game theoretic solutions to general multi-agent reinforcement learning problems.

In this paper we present Meeting Bot, a reinforcement learning based conversational system that interacts with multiple users to schedule meetings. The system is able to interpret user utterences and map them to preferred time slots, which are then fed to a reinforcement learning (RL) system with the goal of converging on an agreeable time slot. The RL system is able to adapt to user preferences and environmental changes in meeting arrival rate while still scheduling effectively. Learning is performed via policy gradient with exploration, by utilizing an MLP as an approximator of the policy function. Results demonstrate that the system outperforms standard scheduling algorithms in terms of overall scheduling efficiency. Additionally, the system is able to adapt its strategy to situations when users consistently reject or accept meetings in certain slots (such as Friday afternoon versus Thursday morning), or when the meeting is called by members who are at a more senior designation.

Policy gradient approaches have gained great success in many complex dynamic decision-making problems, such as the game of Go. However, policy gradient methods suffer from high variance, which implies weak risk control in real applications. Therefore, it is valuable to introduce variance reduction techniques into policy gradient methods to help control the variance in the policy improvement process. Meanwhile, risk-sensitive management in dynamic decision problems is a primary concern in many fields, such as finance and process control. In this paper, we developed a policy search framework for reinforcement learning with variance-related criteria and a variance reduction technique. Our starting point is a standard formulation for the variance of the cost-to-go in episodic tasks. Using this formula, variance-reduced policy search algorithms are proposed. The convergence to local optima of the proposed algorithms is proved, and their applicability is demonstrated on financial-portfolio domains.

Goal recognition is the task of inferring an agent's goals given some or all of the agent's observed actions. Among different ways of problem formulation, goal recognition can be solved as a model-based planning problem using off-the-shell planners. However, obtaining accurate cost or reward models of an agent and incorporating them into the planning model becomes an issue in real applications. Towards this end, we propose an Inverse Reinforcement Learning (IRL)-based opponent behavior modeling method, and apply it in the goal recognition assisted Dynamic Local Network Interdiction (DLNI) problem. We first introduce the overall framework and the DLNI problem domain of our work. After that, an IRL-based human behavior modeling method and Markov Decision Process-based goal recognition are introduced. Experimental results indicate that our learned behavior model has a higher tracking accuracy and yields better interdiction outcomes than other models.

We introduce a novel mechanism for knowledge transfer via concept formation to augment reinforcement learning agents operating in complex, uncertain domains. Based on their observations, agents form concepts and associate them with actions to generalize their decisions at higher levels of abstraction. Concepts serve as simple, portable, efficient packets of hierarchical information that can be learned in parallel. The use of conceptual knowledge simultaneously provides an interpretable, semantic explanation of an agent's decisions, making the techniques promising for human-interaction domains such as games, where human observers wish to inspect an agent's rationale. This technique extends previous work on probabilistic learning with Markov decision processes (MDPs) by introducing rich hierarchical feature structures that can be learned from experience, enabling more effective learning transfer to new, related tasks.

Non-player characters (NPCs) in video games are a common form of frustration for players because they generally provide no explanations for their actions or provide simplistic explanations using fixed scripts. Motivated by this, we consider a new design for agents that can learn about their environments, accomplish a range of goals, and explain what they are doing to a supervisor. We propose a framework for studying this type of agent, and compare it to existing reinforcement learning and self-motivated agent frameworks. We propose a novel design for an initial agent that acts within this framework. Finally, we describe an evaluation centered around the supervisor's satisfaction and understanding of the agent's behavior.

Concept drift - a change, either sudden or gradual, in the underlying properties of data - is one of the most prevalent challenges to maintaining high-performing learned models over time in autonomous systems. In the face of concept drift, one can hope that the old model is sufficiently representative of the new data despite the concept drift, one can discard the old data and retrain a new model with (often limited) new data, or one can use transfer learning methods to combine the old data with the new to create an updated model. Which of these three options is chosen affects not only near-term decisions, but also future needs to transfer or retrain. In this paper, we thus model response to concept drift as a sequential decision making problem and formally frame it as a Markov Decision Process. Our reinforcement learning approach to the problem shows promising results on one synthetic and two real-world datasets.