Freedman, Richard G.


User Interfaces and Scheduling and Planning: Workshop Summary and Proposed Challenges

AAAI Conferences

The User Interfaces and Scheduling and Planning (UISP) Workshop had its inaugural meeting at the 2017 International Conference on Automated Scheduling and Planning (ICAPS). The UISP community focuses on bridging the gap between automated planning and scheduling technologies and user interface (UI) technologies. Planning and scheduling systems need UIs, and UIs can be designed and built using planning and scheduling systems. The workshop participants included representatives from government organizations, industry, and academia with various insights and novel challenges. We summarize the discussions from the workshop as well as outline challenges related to this area of research, introducing the now formally established field to the broader user experience and artificial intelligence communities.


Does the Human's Representation Matter for Unsupervised Activity Recognition?

AAAI Conferences

Unsupervised activity recognition introduces the opportunity for more robust interaction experiences with machines because the human is not limited to only acting with respect to a training dataset. Many approaches currently use latent variable models that have been well studied and developed by the natural language research communities. However, these models are simply used as-is or with minor tweaks on datasets that present an analogy between sensor reading sequences and text documents. Although words have well-defined semantics so that the learned clusters can be interpreted and verified, this is not often the case for sensor readings. For example, novel data from new human activities need to be classified, which relies on the learned clusters; so how does one confirm that new activities are being correctly processed by a robot for interaction? We present several ways that motion capture information can be represented for use in these methods, and then illustrate how the representation choice has the potential to produce variations in the learned clusters.



Integration of Planning with Recognition for Responsive Interaction Using Classical Planners

AAAI Conferences

Interaction between multiple agents requires some form of coordination and a level of mutual awareness. When computers and robots interact with people, they need to recognize human plans and react appropriately. Plan and goal recognition techniques have focused on identifying an agent's task given a sufficiently long action sequence. However, by the time the plan and/or goal are recognized, it may be too late for computing an interactive response. We propose an integration of planning with probabilistic recognition where each method uses intermediate results from the other as a guiding heuristic for recognition of the plan/goal in-progress as well as the interactive response. We show that, like the used recognition method, these interaction problems can be compiled into classical planning problems and solved using off-the-shelf methods. In addition to the methodology, this paper introduces problem categories for different forms of interaction, an evaluation metric for the benefits from the interaction, and extensions to the recognition algorithm that make its intermediate results more practical while the plan is in progress.


Safety in AI-HRI: Challenges Complementing User Experience Quality

AAAI Conferences

Contemporary research in human-robot interaction (HRI) predominantly focuses on the user's experience while controlling a robot. However, with the increased deployment of artificial intelligence (AI) techniques, robots are quickly becoming more autonomous in both academic and industrial experimental settings. In addition to improving the user's interactive experience with AI-operated robots through personalization, dialogue, emotions, and dynamic behavior, there is also a growing need to consider the safety of the interaction. AI may not account for the user's less likely responses, making it possible for an unaware user to be injured by the robot if they have a collision. Issues of trust and acceptance may also come into play if users cannot always understand the robot's thought process, creating a potential for emotional harm. We identify challenges that will need to be addressed in safe AI-HRI and provide an overview of approaches to consider for them, many stemming from the contemporary research.


Integrating Planning and Recognition to Close the Interaction Loop

AAAI Conferences

In many real-world domains, the presence of machines is becoming more ubiquitous to the point that they are usually more than simple automation tools. As part of the environment amongst human users, it is necessary for these computers and robots to be able to interact with them reasonably by either working independently around them or participating in a task, especially one with which a person needs help. This interactive procedure requires several steps: recognizing the user and environment from sensor data, interpreting the user’s activity and motives, determining a responsive behavior, performing the behavior, and then recognizing everything again to confirm the behavior choice and replan if necessary. At the moment, the research areas addressing these steps, activity recognition, plan recognition, intent recognition, and planning, have all been primarily studied independently. However, pipelining each independent process can be risky in real-time situations where there may be enough time to only run a few steps. This leads to a critical question: how do we perform everything under time constraints? In this thesis summary, I propose a framework that integrates these processes by taking advantage of features shared between them.


Integration of Planning with Plan Recognition Using Classical Planners (Extended Abstract)

AAAI Conferences

In order for robots to interact with humans in the world around them, it is important that they are not just aware of the presence of people, but also able to understand what those people are doing. In particular, interaction involves multiple agents which requires some form of coordination, and this cannot be achieved by acting blindly. The field of plan recognition (PR) studies methods for identifying an observed agent’s task or goal given her action sequence. This is often regarded as the inverse of planning which, given a set of goal conditions, aims to derive a sequence of actions that will achieve the goals when performed from a given initial state. Ram´ırez and Geffner (2009; 2010) proposed a simple transformation of PR problems into classical planning problems for which off-the-shelf software is available for quick and efficient implementations. However, there is a reliance on the observed agent’s optimality which makes this PR technique most useful as a post-processing step when some of the final actions are observed. In human-robot interaction (HRI), it is usually too late to interact once the humans are finished performing their tasks. In this paper, we describe ongoing work two extensions to make classical planning-based PR more applicable to the field of HRI. First, we introduce a modification to their algorithm that reduces the optimality bias’s effect so that long-term goals may be recognized at earlier observations. This is then followed by methods for extracting information from these predictions so that the observing agent may run a second pass of the planner to determine its own actions to perform for a fully interactive system.


Temporal and Object Relations in Unsupervised Plan and Activity Recognition

AAAI Conferences

We consider ways to improve the performance of unsupervised plan and activity recognition techniques by considering temporal and object relations in addition to postural data. Temporal relationships can help recognize activities with cyclic structure and are often implicit because plans have degrees of ordering actions. Relations with objects can help disambiguate observed activities that otherwise share a user's posture and position. We develop and investigate graphical models that extend the popular latent Dirichlet allocation approach with temporal and object relations, examine the relative performance and runtime trade-offs using a standard dataset, and consider the cost/benefit trade-offs these extensions offer in the context of human-robot and humancomputer interaction.


Plan and Activity Recognition from a Topic Modeling Perspective

AAAI Conferences

We examine new ways to perform plan recognition (PR) using natural language processing (NLP) techniques. PR often focuses on the structural relationships between consecutive observations and ordered activities that comprise plans. However, NLP commonly treats text as a bag-of-words, omitting such structural relationships and using topic models to break down the distribution of concepts discussed in documents. In this paper, we examine an analogous treatment of plans as distributions of activities. We explore the application of Latent Dirichlet Allocation topic models to human skeletal data of plan execution traces obtained from a RGB-D sensor. This investigation focuses on representing the data as text and interpreting learned activities as a form of activity recognition (AR). Additionally, we explain how the system may perform PR. The initial empirical results suggest that such NLP methods can be useful in complex PR and AR tasks.