Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting Deep Recurrent Q-Network (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.

A commitment represents an agent's intention to attempt to bring about some state of the world that is desired by some agent (possibly itself) in the future. Thus, by making a commitment, an agent is agreeing to make sequential decisions that it believes can cause the desired state to arise. In general, though, an agent's actions will have uncertain outcomes, and thus reaching the desired state cannot be guaranteed. For such sequential decision settings with uncertainty, therefore, commitments can only be probabilistic. We argue that standard notions of commitment are insufficient for probabilistic commitments, and propose a new semantics that judges commitment fulfillment not in terms of whether the agent achieved the desired state, but rather in terms of whether the agent made sequential decisions that in expectation would have achieved the desired state with (at least) the promised probability. We have devised various algorithms that operationalize our semantics, to capture problem contexts with probabilistic commitments arising because action outcomes are uncertain, as well as arising because an agent might realize over time that it does not want to fulfill the commitment.

Multi-robot systems are an exciting application domain for AI research and Dec-POMDPs, specifically. MacDec-POMDP methods can produce high-quality general solutions for realistic heterogeneous multi-robot coordination problems by automatically generating control and communication policies, given a model. In contrast to most existing multi-robot methods that are specialized to a particular problem class, our approach can synthesize policies that exploit any opportunities for coordination that are present in the problem, while balancing uncertainty, sensor information, and information about other agents.

Partially observable stochastic games (POSGs) are a robust and precise model for decentralized decision making under conditions of imperfect information, and extend popular Markov decision problem models. Complexity results for a wide range of such problems are known when agents work cooperatively to pursue common interests. When agents compete, things are less well understood. We show that under one understanding of rational competition, such problems are complete for the class NEXP^NP. This result holds for any such problem comprised of two competing teams of agents, where teams may be of any size whatsoever.

External factors are hard to model using a Markovian state in several real-world planning domains. Although planning can be difficult in such domains, it may be possible to exploit long-term dependencies between states of the environment during planning. We introduce weighted state scenarios to model long-term sequences of states, and we use a model based on a Partially Observable Markov Decision Process to reason about scenarios during planning. Experiments show that our model outperforms other methods for decision making in two real-world domains.

This work presents a general-purpose character reasoning model intended for usage by autonomous team actors that are acting as believable characters (e.g., human team actors fall into this category). The idea is that selecting a cast of believable characters can predetermine a solution to an unexpected challenge that the team may be facing in a rescue mission. This approach in certain cases proves more efficient than an alternative approach based on rational decision making and planning, which ignores the question of character believability. This point is illustrated with a simple numerical example in a virtual world paradigm.

This paper details how OntoAgents, language-endowed intelligent agents developed in the OntoAgent framework, assess their confidence in understanding language inputs. It presents scoring heuristics for the following subtasks of natural language understanding: lexical disambiguation and the establishment of semantic dependencies; reference resolution; nominal compounding; the treatment of fragments; and the interpretation of indirect speech acts. The scoring of confidence in individual linguistic subtasks is a prerequisite for computing the overall confidence in the understanding of an utterance. This, in turn, is a prerequisite for the agent’s deciding how to act upon that level of understanding.

The topic of the self-confidence of autonomous systems is discussed from the perspective of its use in a military environment. The concepts of autonomy and self-confidence are quite different in a military environment from the civilian environment. The military’s recruit indoctrination provided a basis for the concept, the factors affecting the concept, and its measurement and communication. These and other aspects of the topic self-confidence in autonomous systems are discussed along with examples based on current research on the interface between human operators and such systems.

Conventionally, the process of learning the model (exploration) is initialized as either an uninformed or informed policy, where the latter leverages observations to guide future exploration. Informed exploration is ideal as it may allow a model to be learned in fewer samples. However, informed exploration cannot be implemented from the onset when a-priori knowledge on the sensing domain statistics are not available; such policies would only sample the first set of locations, repeatedly. Hence, we present a theoretically-derived bound for transitioning from uninformed exploration to informed exploration for unstructured real-world environments which may be partially-observable and time-varying. This bound is used in tandem with a sparsified Bayesian nonparametric Poisson Exposure Process, which is used to learn to predict the value of information in partiallyobservable and time-varying domains. The result is an uninformed-to-informed exploration policy which outperforms baseline algorithms in real-world data-sets.

Personalized medicine targets the customization of treatment strategies to patients' individual characteristics. Here we consider the problem of optimizing personalized pharmacological treatment strategies for cancer. We focus primarily on developing effective strategies to collect the data necessary for the construction of personalized treatments. We formulate this problem as a contextual bandit and present a new algorithm based on repeated sub-sampling for robust data collection in this framework. We present a case study showing experiments on a simulation setting, built from real data collected in a previous animal experiments. Promising results in this case study have since lead us to deploy this strategy in a partner wet lab to allocate treatments for the next phase of animal experiments.