Websites that encourage consumers to research, rate, and review products online have become an increasingly important factor in purchase decisions. This increased importance has been accompanied by a growth in deceptive opinion spam - fraudulent reviews written with the intent to sound authentic and mislead consumers. In this study, we pool deceptive reviews solicited through crowdsourcing with actual reviews obtained from product review websites. We then explore several human- and machine-based assessment methods to spot deceptive opinion spam in our pooled review set. We find that the combination of human-based assessment methods with easily-obtained statistical information generated from the review text outperforms detection methods using human assessors alone.

Design tasks are notoriously difficult, because success is defined by the perception of the target audience, whose feedback is usually not available during design stages. Commonly, design is performed by professionals who have specific domain knowledge (i.e., an intuitive understanding of the implicit requirements of the task) and do not need the feedback of the perception of the viewers during the process. In this paper, we present a novel design methodology for creating minimal sketches of objects that uses an iterative optimization scheme. We define minimality for a sketch via the minimal number of straight line segments required for correct recognition by 75% of naiive viewers. Crowd-sourcing techniques allow us to directly include the perception of the audience in the design process. By joining designers and crowds, we are able to create a human computation system that can efficiently optimize sketches without requiring high levels of domain knowledge (i.e., design skills) from any worker.

In this paper, we develop some new Captchas belonging to genre – "CAPTCHAs with a purpose". These CAPTCHAs apart from having its applications serve some useful purpose. reCAPTCHA is one such Captcha developed at Carnegie Mellon University. It helps to digitize books. Another such Captcha is Asirra developed at Microsoft which provides homes for homeless animals. In this paper, we present Time based, Sentence based, Human Emotion based CAPTCHAs which have range of other useful purpose such as measuring reaction time of people, promoting news, general knowledge facts, jokes among people while engaging in routine activities such as checking email. Also, one can be used for conducting online polls on a very large scale. We also showcase a New Game with a Purpose called "Identical Emotions" which helps to assign tags describing emotions depicted by the images, to varied images. It can also be used to serve Emotion Based CAPTCHA. We also present a new scheme which renders attack on CAPTCHAs useless and make old CAPTCHAs reusable and help in using CAPTCHAs which might serve some practical purpose which otherwise might be vulnerable to use. This system also enables to use different "CAPTCHAs with a purpose" in conjunction with each other. At present most websites deploy only a single algorithm reCAPTCHA whose practical purpose is to digitize books, thus is limited to one domain. This system can thus, broaden the application domain of CAPTCHAs.

Such systems humans are much more efficient at abstracting and matching take advantage of human abilities--particularly in vision, visual cues across piece borders based on their content. For natural language, and pattern recognition--to handle example, a person looking at a piece of a shredded document instances and aspects of problems that are difficult for can recognize a letter that is only partially present, and an computers. The ESP game (von Ahn and Dabbish 2008), experienced archaeologist looking at a particular piece of FoldIt (Cooper et al. 2010), and reCAPTCHA (von Ahn et a broken artifact can recognize unique patterns that extend al. 2008) are a few examples of successful systems that draw beyond the fragment. Unfortunately, for a human to find a on human contributors and machine computations to tackle matching piece still requires scanning through the pieces, problems in image labeling, protein folding, and text digitization.

Crowdsourcing utilizes human ability by distributing tasks to a large number of workers. It is especially suitable for solving data clustering problems because it provides a way to obtain a similarity measure between objects based on manual annotations, which capture the human perception of similarity among objects.This is in contrast to most clustering algorithms that face the challenge of finding an appropriate similarity measure for the given dataset. Several algorithms have been developed for crowdclustering that combine partial clustering results, each obtained by annotations provided by a different worker, into a single data partition. However, existing crowd-clustering approaches require a large number of annotations, due to the noisy nature of human annotations, leading to a high computational cost in addition to the large cost associated with annotation. We address this problem by developing a novel approach for crowclustering that exploits the technique of matrix completion. Instead of using all the annotations, the proposed algorithm constructs a partially observed similarity matrix based on a subset of pairwise annotation labels that are agreed upon by most annotators. It then deploys the matrix completion algorithm to complete the similarity matrix and obtains the final data partition by applying a spectral clustering algorithm to the completed similarity matrix. We show, both theoretically and empirically, that the proposed approach needs only a small number of manual annotations to obtain an accurate data partition. In effect, we highlight the trade-off between a large number of noisy crowdsourced labels and a small number of high quality labels.

To ensure quality results from crowdsourced tasks requesters often aggregate worker responses and use one of a plethora of strategies for the process of inferring the correct answer from the set of noisy responses. However, all current models assume prior knowledge of all possible outcomes of the task. While not an unreasonable assumption for tasks that can be posited as multiple-choice questions (e.g. n-ary classification), we observe that many tasks do not naturally fit this paradigm, but instead demand a free-response, generalized, formulation where the outcome space is of infinite size (e.g. audio transcription). We call these tasks open questions. We model open questions with a novel probabilistic graphical model, and design and implement LazySusan, a decision-theoretic controller that dynamically requests responses as necessary in order to infer answers to these tasks. Live experiments on Amazon Mechanical Turk demonstrate the superiority of LazySusan at solving SAT Math questions, eliminating 83.2% of the error and achieving greater net utility compared to the state-of-the-art strategy, majority voting.

With the advent of the increasingly participatory Internet and the growing power of the crowd, “Serious Games” have proven to be a fertile approach for gathering task-specific natural language data at very low cost. In this paper we outline a game we call Doodling, based on the sketch-and-convey metaphor used in the popular board game Pictionary(R), with the goal of generating useful natural language data. We explore whether such a paradigm can be successfully extended for conveying more complex syntactic and semantic constructs than the words or short phrases typically used in the board game. Through a series of user experiments, we show that this is indeed the case, and that valuable parallel language data may be produced as a byproduct. In addition, we explore extensions to this paradigm along two axes – going online (vs. face-to-face) and going cross-lingual. The results in each of the sets of experiments confirm the potential of Doodling game to generate data in large quantities and across languages, and thus provide a new means of developing data sets and technologies for resource-poor languages.

Online labor marketplaces offer the potential to automate a variety of tasks too difficult for computers, but present requesters with significant difficulties in obtaining accurate results. We share experiences from building MobileWorks, a crowd platform that departs from the marketplace model to provide robust, high-quality results. Three architectural contributions yield measurably improved accuracy on input tasks.  A dynamic work routing system identifies expertise in the crowd and ensures that all work posted into the system is completed with bounded completion times and at fair worker prices. A peer management system ensures that incorrect answers are prevented by experienced members of the crowd. Last, social interaction techniques give the best workers the ability and incentives to manage, teach & supervise other members of the crowd, as well as to clarify tasks. This process filters worker error and allows the crowd to collectively learn how to solve unfamiliar tasks. (extended abstract)

A general consensus on representation of emotions and feelings in cognitive architectures is currently missing; yet artificial emotional intelligence is vital for the integration of future robots into the human society. This work introduces one possible approach to representation and processing of emotional mental states and attitudes, that allows for implementation of control of agent behavior by emotions as well as for recognition of emotional motivations in another agent's behavior. One particular advantage of this approach is that it allows for representation and processing of complex/social emotional attitudes, like shame, jealousy, resentment, or humor. The proposed validation of the approach is based on simulation of the emergence of emotional relationships in a small group of agents in a virtual environment.

Robust methods for representing, generalizing, and sharing knowledge across various robotics systems and configurations are important in many domains of robotics research and application. In this paper we present a method for modeling tasks and robot skills to simplify the programming and reuse of knowledge between robots in manufacturing environments. Specifically, we propose an assembly taxonomy designed to represent the decomposition of high-level, complex assembly tasks into simple skills and skill primitives that the robot must use in a specified sequence. By using programming by demonstration to populate the taxonomy, we propose a method to easily interact with and reuse knowledge in various manufacturing robotics systems, making it possible to reduce programming time and overhead. We present both a detailed discussion of this taxonomy, as well as an example of how the taxonomy can be applied to an assembly task.