The problem of learning to predict structured labels is of key importance in many applications. However, for general graph structure both learning and inference in this setting are intractable. Here we show that it is possible to circumvent this difficulty when the input distribution is rich enough via a method similar in spirit to pseudo-likelihood. We show how our new method achieves consistency, and illustrate empirically that it indeed performs as well as exact methods when sufficiently large training sets are used.

Bayesian approaches to preference elicitation (PE) are particularly attractive due to their ability to explicitly model uncertainty in users' latent utility functions. However, previous approaches to Bayesian PE have ignored the important problem of generalizing from previous users to an unseen user in order to reduce the elicitation burden on new users. In this paper, we address this deficiency by introducing a Gaussian Process (GP) prior over users' latent utility functions on the joint space of user and item features. We learn the hyper-parameters of this GP on a set of preferences of previous users and use it to aid in the elicitation process for a new user. This approach provides a flexible model of a multi-user utility function, facilitates an efficient value of information (VOI) heuristic query selection strategy, and provides a principled way to incorporate the elicitations of multiple users back into the model. We show the effectiveness of our method in comparison to previous work on a real dataset of user preferences over sushi types.

In discriminative machine learning one is interested in training a system to optimize a certain desired measure of performance, or loss. In binary classification one typically tries to minimizes the error rate. But in structured prediction each task often has its own measure of performance such as the BLEU score in machine translation or the intersection-over-union score in PASCAL segmentation. The most common approaches to structured prediction, structural SVMs and CRFs, do not minimize the task loss: the former minimizes a surrogate loss with no guarantees for task loss and the latter minimizes log loss independent of task loss. The main contribution of this paper is a theorem stating that a certain perceptron-like learning rule, involving features vectors derived from loss-adjusted inference, directly corresponds to the gradient of task loss. We give empirical results on phonetic alignment of a standard test set from the TIMIT corpus, which surpasses all previously reported results on this problem.

This paper introduces a principled approach for the design of a scalable general reinforcement learning agent. Our approach is based on a direct approximation of AIXI, a Bayesian optimality notion for general reinforcement learning agents. Previously, it has been unclear whether the theory of AIXI could motivate the design of practical algorithms. We answer this hitherto open question in the affirmative, by providing the first computationally feasible approximation to the AIXI agent. To develop our approximation, we introduce a new Monte-Carlo Tree Search algorithm along with an agent-specific extension to the Context Tree Weighting algorithm. Empirically, we present a set of encouraging results on a variety of stochastic and partially observable domains. We conclude by proposing a number of directions for future research.

We present a probabilistic logic programming framework to reinforcement learning, by integrating reinforce-ment learning, in POMDP environments, with normal hybrid probabilistic logic programs with probabilistic answer set seman-tics, that is capable of representing domain-specific knowledge. We formally prove the correctness of our approach. We show that the complexity of finding a policy for a reinforcement learning problem in our approach is NP-complete. In addition, we show that any reinforcement learning problem can be encoded as a classical logic program with answer set semantics. We also show that a reinforcement learning problem can be encoded as a SAT problem. We present a new high level action description language that allows the factored representation of POMDP. Moreover, we modify the original model of POMDP so that it be able to distinguish between knowledge producing actions and actions that change the environment.

Often we wish to predict a large number of variables that depend on each other as well as on other observed variables. Structured prediction methods are essentially a combination of classification and graphical modeling, combining the ability of graphical models to compactly model multivariate data with the ability of classification methods to perform prediction using large sets of input features. This tutorial describes conditional random fields, a popular probabilistic method for structured prediction. CRFs have seen wide application in natural language processing, computer vision, and bioinformatics. We describe methods for inference and parameter estimation for CRFs, including practical issues for implementing large scale CRFs. We do not assume previous knowledge of graphical modeling, so this tutorial is intended to be useful to practitioners in a wide variety of fields.

Traditional dialogue management systems are tightly coupled with the sensing ability of a single computer. How to organize an interaction in pervasive environments to provide a friendly and integrated interface to users is an important issue. This requires a transition of the human-computer interaction (HCI) from tight coupling to loose coupling. This paper proposes a hierarchical multimodal framework for pervasive interactions. Our system is designed to remind the activities of daily living for individuals with cognitive impairments.The system is composed of Markov decision processes for activity planing, and multimodal partially observable Markov decision processes for action planning and executing. Empirical results demonstrate the hierarchical multimodal framework establishes a flexible mechanism for pervasive interaction systems.

Our learning by teaching environment has students take on the role and responsibilities of a teacher to a virtual student named Betty. The environment is structured so that successfully instructing their teachable agent requires the students to learn and understand science topics for themselves. This process is supported by adaptive scaffolding and feedback from the system. This feedback is instantiated through the interactions with the teachable agent and a mentor agent, named Mr. Davis. This paper provides an overview of two studies that were conducted with 5th grade science students and a description of the analysis techniques that we have developed for interpreting students’ activities in this learning environment.

A promising application area for proactive assistant agents is automated tutoring and training.  Intelligent tutoring systems (ITSs) assist tutors and tutees by automating diagnosis and adaptive tutoring. These tasks are well modeled by a partially observable Markov decision process (POMDP) since it accounts for the uncertainty inherent in diagnosis. However, an important aspect of making POMDP solvers feasible for real-world problems is selecting appropriate representations for states, actions, and observations. This paper studies two scalable POMDP state and observation representations. State queues allow POMDPs to temporarily ignore less-relevant states. Observation chains represent information in independent dimensions using sequences of observations to reduce the size of the observation set. Preliminary experiments with simulated tutees suggest the experimental representations perform as well as lossless POMDPs, and can model much larger problems.

A new architectural framework for a metacognitive tutoring system is presented that is aimed to stimulate self-regulatory behavior in the learner.The new framework extends the cognitive architecture of TutorJ that has been already proposed by some of the authors. TutorJ relies mainly on dialogic interaction with the user, and makes use of a statistical dialogue planner implemented through a Partially Observable Markov Decision Process (POMDP). A suitable two-level structure has been designed for the statistical reasoner to cope with measuring and stimulating metacognitive skills in the user. Suitable actions have been designed to this purpose starting from the analysis of the main questionnaires proposed in the literature. Our reasoner has been designed to model the relation between each item in a questionnaire and the related metacognitive skill, so the proper action can be selected by the tutoring agent. The complete framework is detailed, the reasoner structure is discussed, and a simple application scenario is presented.