We consider active maximum a posteriori (MAP) inference problem for Hidden Markov Models (HMM), where, given an initial MAP estimate of the hidden sequence, we select to label certain states in the sequence to improve the estimation accuracy of the remaining states. We develop an analytical approach to this problem for the case of binary symmetric HMMs, and obtain a closed form solution that relates the expected error reduction to model parameters under the specified active inference scheme. We then use this solution to determine most optimal active inference scheme in terms of error reduction, and examine the relation of those schemes to heuristic principles of uncertainty reduction and solution unicity.

We give an overview of how the CADENCE architecture addresses the problem of turn-taking in embodied interaction.

We describe a novel system for discovering and characterizing emerging events. We define event emergence to be a developing situation comprised of a series of sub-events. To detect sub-events from a very large, continuous textual input stream, we use two techniques: (1) frequency-based detection of sub-events that are potentially entailed by an emerging event; and (2) anomaly-based detection of other sub-events that are potentially indicative of an emerging event. Identifying emerging events from detected sub-events involves connecting sub-events to each other and to the relevant emerging events within the event models and estimating the likelihood of possible emerging events. Each sub-event can be part of a number of emerging events and supports various event models to varying degrees. We adopt a coherent and compact model that probabilistically identifies emerging events. The innovative aspect of our work is a well-defined framework where statistical Big Data techniques are informed by event semantics and inference techniques (and vice versa). Our work is strongly grounded in semantics and knowledge representation, which enables us to produce more reliable results than would otherwise be possible with a purely statistical approach.

The goal of transfer is to use knowledge obtained by solving one task to improvea robot's (or software agent's) performance in future tasks. In general, we do not expect this to work; for transfer to be feasible, there must be something in common between the source task(s) and goal task(s). The question at the core of the transfer learning enterprise is therefore: what makes two tasks related?, or more generally, how do you define a family of related tasks? Given a precise definition of how a particular family of tasks is related, we can formulate clear optimizationmethods for selecting source tasks and determining what knowledge should be imported from the source task(s), and how it should be used in the target task(s). This paper describes one model that has appeared in several different research scenarios where an agent is faced with afamily of tasks that have similar, but not identical, dynamics (or reward functions). For example, a human learning to play baseball may, over the course of their career,be exposed to several different bats, each with slightly different weights and lengths.A human who has learned to play baseball well with one bat would be expected to be able to pick up any similar bat and use it.Similarly, when learning to drive a car, one may learn in more than one car, and then be expected to be able to drive any make and model of car (within reasonablevariations) with little or no relearning. These examples are instances of exactly the kind of flexible, reliable,and sample-efficient behavior that we should be aiming to achieve in robotics applications. One way to model such a family of tasks is to posit that they are generated by asmall set of latent parameters (e.g., the length and weight of the bat, or parametersdescribing the various physical properties of the car's steering system and clutch) thatare fixed for each problem instance (e.g., for each bat, or car), but are not directlyobservable by the agent. Defining a distributionover these latent parameters results in a family of related tasks, and transferis feasible to the extent that the number of latent variables is small, the task dynamics(or reward function) vary smoothly with them, and to the extent to which they can eitherbe ignored or identified using transition data from the task.This model has appeared under several different names in the literature; we refer to it as a hidden-parameterMarkov decision process (or HIP-MDP).

We present ongoing research to develop novel option discovery methods for complex domains that are represented as Object-Oriented Markov Decision Processes (OO-MDPs) (Diuk, Cohen, and Littman, 2008). We describe Portable Multi-policy Option Discovery for Automated Learning (P-MODAL), an initial framework that extends Pickett and Barto’s (2002) PolicyBlocks approach to OO-MDPs. We also discuss future work that will use additional representations and techniques to handle scalability and learning challenges.

Robotic agents often map perceptual input to simplified representations that do not reflect the complexity and richness of the world. This simplification is due in large part to the limitations of planning algorithms, which fail in large stochastic state spaces on account of the well-known "curse of dimensionality." Existing approaches to address this problem fail to prevent autonomous agents from considering many actions which would be obviously irrelevant to a human solving the same problem. We formalize the notion of affordances as knowledge added to an Markov Decision Process (MDP) that prunes actions in a state- and reward- general way. This pruning significantly reduces the number of state-action pairs the agent needs to evaluate in order to act near-optimally. We demonstrate our approach in the Minecraft domain as a model for robotic tasks, showing significant increase in speed and reduction in state-space exploration during planning. Further, we provide a learning framework that enables an agent to learn affordances through experience, opening the door for agents to learn to adapt and plan through new situations. We provide preliminary results indicating that the learning process effectively produces affordances that help solve an MDP faster, suggesting that affordances serve as an effective, transferable piece of knowledge for planning agents in large state spaces.

Research on POMDP formulations for collaborative tasks in game AI applications (Nguyen et al. 2011; Macindoe, The development of new industrial robotic systems that operate Kaelbling, and Lozano-Pérez 2012; Silver and Veness in the same physical space as people highlights the 2010) also assumed a known human model. Additionally, emerging need for robots that can integrate seamlessly into previous partially observable formalisms (Ong et al. 2010; human group dynamics by adapting to the personalized style Bandyopadhyay et al. 2013; Broz, Nourbakhsh, and Simmons of human teammates. This adaptation requires learning a statistical 2011; Fern and Tadepalli 2010; Nguyen et al. 2011; model of human behavior and integrating this model Macindoe, Kaelbling, and Lozano-Pérez 2012) in assistive into the decision-making algorithm of the robot in a principled or collaborative tasks represented the preference or intention way. We present a framework for automatically learning of the human for their own actions, rather than those of human user models from joint-action demonstrations the robot, as the partially observable variable.

Vocalics is the study of the nonverbal aspects of speech, such as volume, pitch, and rate. Our contribution is a parametric We present an overview of the control, recognition, decision-making, vocalic behavior controller that autonomously adjusts and learning techniques utilized by the Interaction the robot speaker volume based on models of how a Lab (robotics.usc.edu/interaction) at the University human user will hear speech produced by the robot. These of Southern California (USC) to enable autonomy in sociable models vary with distance, orientation, and perceived environmental and socially assistive robots. These techniques are implemented interference (Mead & Matarić 2014). Our future with two software libraries: 1) the Social Behavior work will investigate adapting the pitch and rate of speech Library (SBL) provides autonomous social behavior produced by a robot to improve user speech perception.

Robots that are integrated into day-to-day settings as assistants, collaborators, and companions will engage in dynamic, physically-situated social interactions with their users. Enabling such interactions will require appropriate models and representations for interaction. In this paper, we argue that the dynamic, physically-situated interactions between humans and robots can be characterized as a system of distributed cognition, that this system can be represented using probabilistic graphical models (PGMs), and that the parameters of these models can be learned from human interactions. We illustrate the application of this perspective in our ongoing research on modeling dyadic referential communication.

In order to successfully interact with multiple humans in social situations, an intelligent robot should have the ability to track multi-humans, and understand their motion intentions. We formalize this problem as a hidden Markov model, and estimate the posterior densities by particle filtering over sets approach. Our approach avoids directly performing observation-to-target association by defining a set as a joint state. The human identification problem is then solved in an expectation-maximization way. We evaluate the effectiveness of our approach by both benchamark test and real robot experiments.