We develop a flexible reward plan to elicit truthful predictive probability distribution over a set of uncertain events from workers.  In our reward plan, the principal can assign rewards for incorrect predictions according to her similarity between events.  In the spherical proper scoring rule, a worker's expected utility is represented as the inner product of her truthful predictive probability and her declared probability. We generalize the inner product by introducing a reward matrix that defines a reward for each prediction-outcome pair. We show that if the reward matrix is symmetric and positive definite, the spherical proper scoring rule guarantees the maximization of a worker's expected utility when she truthfully declares her prediction.

This paper introduces the Tropel system which enables non-technical users to create arbitrary visual detectors without first annotating a training set. Our primary contribution is a crowd active learning pipeline that is seeded with only a single positive example and an unlabeled set of training images. We examine the crowd's ability to train visual detectors given severely limited training themselves. This paper presents a series of experiments that reveal the relationship between worker training, worker consensus and the average precision of detectors trained by crowd-in-the-loop active learning. In order to verify the efficacy of our system, we train detectors for bird species that work nearly as well as those trained on the exhaustively labeled CUB 200 dataset at significantly lower cost and with little effort from the end user. To further illustrate the usefulness of our pipeline, we demonstrate qualitative results on unlabeled datasets containing fashion images and street-level photographs of Paris.

The present study explores a method to reduce abusive worker behavior on Amazon Mechanical Turk (AMT), namely reminding workers of moral standards. We manipulated workers’ awareness of moral standards via the presence or the absence of an honesty statement in a survey. The results showed that the honesty statement significantly improved workers’ performance during the first half of the survey. This suggests that a moral reminder is a simple and efficient way to reduce abusive worker behavior in a relatively short survey on AMT.

Artificial Intelligence (AI) techniques have been widely used in video games to control non-playable characters. More recently, AI has been applied to automated story generation with the objective of managing the player's experience in an interactive narrative. Such AI experience managers can generate and adapt narrative dynamically, often in response to the player's in-game actions. We implement and evaluate a recently proposed AI experience manager, PACE, which predicts the player's emotional response to a narrative event and uses such predictions to shape the narrative to keep the player on an author-supplied target emotional curve.

Mobile digital games are dominantly released under the freemium business model, but only a small fraction of the players makes any purchases. The ability to predict who will make a purchase enables optimization of marketing efforts, and tailoring customer relationship management to the specific user's profile. Here this challenge is addressed via two models for predicting purchasing players, using a 100,000 player dataset: 1) A classification model focused on predicting whether a purchase will occur or not. 2) a regression model focused on predicting the number of purchases a user will make. Both models are presented within a decision and regression tree framework for building rules that are actionable by companies. To the best of our knowledge, this is the first study investigating purchase decisions in freemium mobile products from a user behavior perspective and adopting behavior-driven learning approaches to this problem.

Real-Time Strategy (RTS) games have shown to be very resilient to standard  adversarial tree search techniques. Recently, a few approaches to tackle their complexity have emerged that use game state or move abstractions, or both. Unfortunately, the supporting experiments were either limited to simpler RTS environments ( u RTS, SparCraft) or lack testing against state-of-the-art game playing agents. Here, we propose Puppet Search , a new adversarial search framework based on scripts that can expose choice points to a look-ahead search procedure. Selecting a combination of a script and decisions for its choice points represents a move to be applied next. Such moves can be executed in the actual game, thus letting the script play, or in an abstract representation of the game state which can be used by an adversarial tree search algorithm. Puppet Search returns a principal variation of scripts and choices to be executed by the agent for a given time span. We implemented the algorithm in a complete StarCraft bot. Experiments show that it matches or outperforms all of the individual scripts that it uses when playing against state-of-the-art bots from the 2014 AIIDE StarCraft competition.

Counting objects is a fundamental image processisng primitive, and has many scientific, health, surveillance, security, and military applications. Existing supervised computer vision techniques typically require large quantities of labeled training data, and even with that, fail to return accurate results in all but the most stylized settings. Using vanilla crowdsourcing, on the other hand, can lead to significant errors, especially on images with many objects. In this paper, we present our JellyBean suite of algorithms, that combines the best of crowds and computer vision to count objects in images, and uses judicious decomposition of images to greatly improve accuracy at low cost. Our algorithms have several desirable properties: (i) they are theoretically optimal or near-optimal , in that they ask as few questions as possible to humans (under certain intuitively reasonable assumptions that we justify in our paper experimentally); (ii) they operate under stand-alone or hybrid modes, in that they can either work independent of computer vision algorithms, or work in concert with them, depending on whether the computer vision techniques are available or useful for the given setting; (iii) they perform very well in practice, returning accurate counts on images that no individual worker or computer vision algorithm can count correctly, while not incurring a high cost.

Dunyazad is a system which creates narrative choices à la Choose-Your-Own-Adventure books. It attempts to generate choices that achieve specific poetic effects. This paper demonstrates Dunyazad’s ability to manage player expectations by having it generate three distinct choice structures: obvious choices, relaxed choices, and dilemmas. Using answer set programming, Dunyazad’s choice generation system directly encodes a theory of choice poetics, so flaws in its output can inform both the system and the theory itself. Survey data presented here thus not only validate that players’ perceptions match Dunyazad’s intentions, but also have implications for the theory of choice poetics. Statistical analysis of our data indicates that Dunyazad can successfully construct obvious choices, relaxed choices, and dilemmas.

While recent work has shown that a worker’s performance can be more accurately modeled by temporal correlation in task performance, a fundamental challenge remains in the need for expert gold labels to evaluate a worker’s performance. To solve this problem, we explore two methods of utilizing limited gold labels, initial training and periodic updating. Furthermore, we present a novel way of learning a prediction model in the absence of gold labels with uncertaintyaware learning and soft-label updating. Our experiment with a real crowdsourcing dataset demonstrates that periodic updating tends to show better performance than initial training when the number of gold labels are very limited (< 25).

Smart decision making at the tactical level is important for Artificial Intelligence (AI) agents to perform well in the domain of real-time strategy (RTS) games. Winning battles is crucial in RTS games, and while humans can decide when and how to attack based on their experience, it is challenging for AI agents to estimate combat outcomes accurately. A few existing models address this problem in the game of StarCraft but present many restrictions, such as not modeling injured units, supporting only a small number of unit types, or being able to predict the winner of a fight but not the remaining army. Prediction using simulations is a popular method, but generally slow and requires extensive coding to model the game engine accurately. This paper introduces a model based on Lanchester's attrition laws which addresses the mentioned limitations while being faster than running simulations. Unit strength values are learned using maximum likelihood estimation from past recorded battles. We present experiments that use a StarCraft simulator for generating battles for both training and testing, and show that the model is capable of making accurate predictions. Furthermore, we implemented our method in a StarCraft bot that uses either this or traditional simulations to decide when to attack or to retreat. We present tournament results (against top bots from 2014 AIIDE competition) comparing the performances of the two versions, and show increased winning percentages for our method.