In this work we present a technique to use natural language to help reinforcement learning generalize to unseen environments. This technique uses neural machine translation, specifically the use of encoder-decoder networks, to learn associations between natural language behavior descriptions and state-action information. We then use this learned model to guide agent exploration using a modified version of policy shaping to make it more effective at learning in unseen environments. We evaluate this technique using the popular arcade game, Frogger, under ideal and non-ideal conditions. This evaluation shows that our modified policy shaping algorithm improves over a Q-learning agent as well as a baseline version of policy shaping.

Automated story generation is the problem of automatically selecting a sequence of events, actions, or words that can be told as a story. We seek to develop a system that can generate stories by learning everything it needs to know from textual story corpora. To date, recurrent neural networks that learn language models at character, word, or sentence levels have had little success generating coherent stories. We explore the question of event representations that provide a mid-level of abstraction between words and sentences in order to retain the semantic information of the original data while minimizing event sparsity. We present a technique for preprocessing textual story data into event sequences. We then present a technique for automated story generation whereby we decompose the problem into the generation of successive events (event2event) and the generation of natural language sentences from events (event2sentence). We give empirical results comparing different event representations and their effects on event successor generation and the translation of events to natural language.

In this work we introduce Quixote, a system that makes programming virtual agents more accessible to non-programmers by enabling these agents to be trained using the sociocultural knowledge present in stories. Quixote uses a corpus of exemplar stories to automatically engineer a reward function that is used to train virtual agents to exhibit desired behaviors using reinforcement learning. We show the effectiveness of our system with a case study conducted in a virtual environment called Robbery World that simulates a bank robbery scenario. In this case study, we use a corpus of stories crowdsourced from Amazon Mechanical Turk to guide learning. We evaluate Quixote under a variety of different conditions to determine the overall effectiveness of the system in Robbery World.

Value alignment is a property of an intelligent agent indicating that it can only pursue goals that are beneficial to humans. Successful value alignment should ensure that an artificial general intelligence cannot intentionally or unintentionally perform behaviors that adversely affect humans. This is problematic in practice since it is difficult to exhaustively enumerated by human programmers. In order for successful value alignment, we argue that values should be learned. In this paper, we hypothesize that an artificial intelligence that can read and understand stories can learn the values tacitly held by the culture from which the stories originate.We describe preliminary work on using stories to generate a value-aligned reward signal for reinforcement learning agents that prevents psychotic-appearing behavior.

Bot detection is an emerging problem in social games that requires different approaches from those used in massively multi-player online games (MMOGs). We focus on mouse selections as a key element of bot detection. We hypothesize that certain interface elements result in predictable differences in mouse selections, which we call spatial game signatures, and that those signatures can be used to model player interactions that are specific to the game mechanics and game interface. We performed a study in which users played a game representative of social games. We collected in-game actions, from which we empirically identified these signatures, and show that these signatures result in a viable approach to bot detection. We make three contributions. First, we introduce the idea of spatial game signatures. Second, we show that the assumption that mouse clicks are normally distributed about the center of buttons is not true for every interface element. Finally, we provide methodologies for using spatial game signatures for bot detection.

What features contribute to player enjoyment and player retentionhas been a popular research topic in video games research;however, the question of what causes players to quit agame has received little attention by comparison. In this paper,we examine 5 quantitative features of the game Scrabblesquein order to determine what behaviors are predictors ofa player prematurely ending a game session. We identified afeature transformation that notably improves prediction accuracy.We used a naive Bayes model to determine that there areseveral transformed feature sequences that are accurate predictorsof players terminating game sessions before the endof the game.We also identify several trends that exist in thesesequences to give a more general idea as to what behaviorsare characteristic early indicators of players quitting.

In this paper, we survey techniques used in intelligent tutoring systems (ITSs) to model student knowledge. The three techniques that we review in detail are knowledge tracing, performance factor analysis, and matrix factorization. We also briefly cover other techniques that have been used. This review is meant to be a repository of knowledge for those who want to integrate these techniques into serious games. It is also meant to increase awareness and interest as to the techniques available that can be integrated into serious games.

My research involves creating and evaluating adaptive gameenvironments using player models created using data-driventechniques and algorithms. I hypothesize that I will be able tochange parts of a game to elicit certain behaviors from players,and that these changes will also result in an increase ofengagement and/or intrinsic motivation.

Conflict is an essential element of interesting stories. In previous work, we proposed a formal model of narrative conflict. We also described 7 dimensions which can be used to distinguish one conflict from another: participants, subject, duration, balance, directness, intensity, and resolution. This paper presents the results of an experiment designed to measure how well our metrics for balance, directness, intensity, and resolution predict the responses of human readers when asked to measure these same values in a set of four stories. We conclude that our metrics are able to rank stories similarly to human readers.