Technology
Preface
Felner, Ariel (Ben Gurion Univserity of the Negev)
Recently, there has been a growing interest in multiagent path planning (MAPF). Applications include vehicle fleet coordination, computer games, robotics, and various military scenarios. Some researchers have worked at a theoretical level, while others implemented solvers to specific applications. Consequently, similar concepts were developed in different subcommunities, using varying terminology.
Capturing Browsing Interests of Users into Web Usage Profiles
Kabir, Shaily (Concordia University) | Mudur, Sudhir P. (Concordia University) | Shiri, Nematollaah (Concordia University)
We present a new weighted session similarity measure to capture the browsing interests of users in web usage profiles discovered from web log data. We base our similarity measure on the reasonable assumption that when users spend longer times on pages or revisit pages in the same session, then very likely, such pages are of greater interest to the user. The proposed similarity measure combines structural similarity with session-wise page significance. The latter, representing the degree of user interest, is computed using frequency and duration of a page access. Web usage profiles are generated using this similarity measure by applying a fuzzy clustering algorithm to web log data. For evaluating the effectiveness of the proposed measure, we adapt two model-based collaborative filtering algorithms for recommending pages. Experimental results show considerable improvement in overall performance of recommender systems as compared to use of other existing similarity measures.
Preface
Jannach, Dietmar (TU Dortmund)
Thee technical program of this workshop consists of presentations of recent, high-quality research contributions, which were selected by the workshop's international program committee in a peer review process. Five long papers and three short papers were accepted for presentation. The papers address a variety of topics in the context of personalization and recommender systems such as new techniques for group recommendation; user modeling and recommendation on the social web; automated content analysis for personalization and recommendation and mobile advertising.
Learning Sociocultural Knowledge via Crowdsourced Examples
Li, Boyang (Georgia Institute of Technology) | Appling, Darren Scott (Georgia Institute of Technology) | Lee-Urban, Stephen (Georgia Institute of Technology) | Riedl, Mark (Georgia Institute of Technology)
Computational systems can use sociocultural knowledge to understand human behavior and interact with humans in more natural ways. However, such systems are limited by their reliance on hand-authored sociocultural knowledge and models. We introduce an approach to automatically learn robust, script-like sociocultural knowledge from crowdsourced narratives. Crowdsourcing, the use of anonymous human workers, provides an opportunity for rapidly acquiring a corpus of examples of situations that are highly specialized for our purpose yet sufficiently varied, from which we can learn a versatile script. We describe a semi-automated process by which we query human workers to write natural language narrative examples of a given situation and learn the set of events that can occur and the typical even ordering.
Machine-Learning for Spammer Detection in Crowd-Sourcing
Halpin, Harry (W3C/Massachusetts Institute of Technology) | Blanco, Roi (Yahoo! Research)
Over a series of evaluation experiments conducted using naive judges recruited and managed via Amazon's Mechanical Turk facility using a task from information retrieval (IR), we show that a SVM shows itself to have a very high accuracy when the machine-learner is trained and tested on a single task and that the method was portable from more complex tasks to simpler tasks, but not vice versa.
Improving Quality of Crowdsourced Labels via Probabilistic Matrix Factorization
Jung, Hyun Joon (University of Texas at Austin) | Lease, Matthew (University of Texas at Austin)
In crowdsourced relevance judging, each crowd workertypically judges only a small number of examples,yielding a sparse and imbalanced set of judgments inwhich relatively few workers influence output consensuslabels, particularly with simple consensus methodslike majority voting. We show how probabilistic matrixfactorization, a standard approach in collaborative filtering,can be used to infer missing worker judgments suchthat all workers influence output labels. Given completeworker judgments inferred by PMF, we evaluate impactin unsupervised and supervised scenarios. In thesupervised case, we consider both weighted voting andworker selection strategies based on worker accuracy.Experiments on a synthetic data set and a real turk dataset with crowd judgments from the 2010 TREC RelevanceFeedback Track show promise of the PMF approachmerits further investigation and analysis.
Towards Social Norm Design for Crowdsourcing Markets
Ho, Chien-Ju (University of California, Los Angeles) | Zhang, Yu (University of California, Los Angeles) | Vaughan, Jennifer Wortman (University of California, Los Angeles) | Schaar, Mihaela van der (University of California, Los Angeles)
Crowdsourcing markets, such as Amazon Mechanical Turk, provide a platform for matching prospective workers around the world with tasks. However, they are often plagued by workers who attempt to exert as little effort as possible, and requesters who deny workers payment for their labor. For crowdsourcing markets to succeed, it is essential to discourage such behavior. With this in mind, we propose a framework for the design and analysis of incentive mechanisms based on social norms, which consist of a set of rules that participants are expected to follow, and a mechanism for updating participants’ public reputations based on whether or not they do. We start by considering the most basic version of our model, which contains only homogeneous participants and randomly matches workers with tasks. The optimal social norm in this setting turns out to be a simple, easily comprehensible incentive mechanism in which market participants are encouraged to play a tit-for-tat-like strategy. This simple mechanism is optimal even when the set of market participants changes dynamically over time, or when some fraction of the participants may be irrational. In addition to the basic model, we demonstrate how this framework can be applied to situations in which there are heterogeneous users by giving several illustrating examples. This work is a first step towards a complete theory of incentive design for crowdsourcing systems. We hope to build upon this framework and explore more interesting and practical aspects of real online labor markets in our future work.
Part Annotations via Pairwise Correspondence
Maji, Subhransu (Toyota Technological Institute at Chicago) | Shakhnarovich, Gregory (Toyota Technological Institute at Chicago)
We explore the use of an interface to mark pairs of points on two images which are in "correspondence" with one another, as a way of collecting part annotations. The interface allows annotations of visual categories that are structurally diverse, such as chairs and buildings, where it is difficult to define a set of parts, or landmarks, that are consistent, namable or uniquely defined across all instances of the category. It allows flexibility in annotation - the landmarks can be instance specific, are not constrained by language, could be many to one, etc and requires little category specific instructions. We compare our approach to two popular methods of collecting part annotations, (1) drawing bounding boxes for a set of parts, and (2) annotating a set of landmarks, in terms of annotation setup overhead, cost, difficulty, applicability and utility, and identify scenarios where one method is better suited than the others. Preliminary experiments suggest that such annotations between a sparse set of pairs can be used to bootstrap many high level visual recognition tasks such as part discovery and semantic saliency.
TurkServer: Enabling Synchronous and Longitudinal Online Experiments
Mao, Andrew (Harvard University) | Chen, Yiling (Harvard University) | Gajos, Krzysztof Z. (Harvard University) | Parkes, David C. (Harvard University) | Procaccia, Ariel D (Carnegie Mellon University) | Zhang, Haoqi (Harvard University)
With the proliferation of online labor markets and other social computing platforms, online experiments have become a low-cost and scalable way to empirically test hypotheses and mechanisms in both human computation and social science. Yet, despite the potential in designing more powerful and expressive online experiments using multiple subjects, researchers still face many technical and logistical difficulties. We see synchronous and longitudinal experiments involving real-time interaction between participants as a dual-use paradigm for both human computation and social science, and present TurkServer, a platform that facilitates these types of experiments on Amazon Mechanical Turk. Our work has the potential to make more fruitful online experiments accessible to researchers in many different fields.
Contextual Commonsense Knowledge Acquisition from Social Content by Crowd-Sourcing Explanations
Kuo, Yen-Ling (National Taiwan University) | Hsu, Jane Yung-jen (National Taiwan University) | Shih, Fuming (Massachusetts Institute of Technology)
Contextual knowledge is essential in answering questions given specific observations. While recent approaches to building commonsense knowledge basesvia text mining and/or crowdsourcing are successful,contextual knowledge is largely missing. To addressthis gap, this paper presents SocialExplain, a novel approach to acquiring contextual commonsense knowledge from explanations of social content. The acquisition process is broken into two cognitively simple tasks:to identify contextual clues from the given social content, and to explain the content with the clues. An experiment was conducted to show that multiple piecesof contextual commonsense knowledge can be identi-fied from a small number of tweets. Online users verified that 92.45% of the acquired sentences are good,and 95.92% are new sentences compared with existingcrowd-sourced commonsense knowledge bases.