JAG: A Crowdsourcing Framework for Joint Assessment and Peer Grading

AAAI Conferences

Generation and evaluation of crowdsourced content is commonly treated as two separate processes, performed at different times and by two distinct groups of people: content creators and content assessors. As a result, most crowdsourcing tasks follow this template: one group of workers generates content and another group of workers evaluates it. In an educational setting, for example, content creators are traditionally students that submit open-response answers to assignments (e.g., a short answer, a circuit diagram, or a formula) and content assessors are instructors that grade these submissions. Despite the considerable success of peer-grading in massive open online courses (MOOCs), the process of test-taking and grading are still treated as two distinct tasks which typically occur at different times, and require an additional overhead of grader training and incentivization. Inspired by this problem in the context of education, we propose a general crowdsourcing framework that fuses open-response test-taking (content generation) and assessment into a single, streamlined process that appears to students in the form of an explicit test, but where everyone also acts as an implicit grader. The advantages offered by our framework include: a common incentive mechanism for both the creation and evaluation of content, and a probabilistic model that jointly models the processes of contribution and evaluation, facilitating efficient estimation of the quality of the contributions and the competency of the contributors. We demonstrate the effectiveness and limits of our framework via simulations and a real-world user study.


Distributed Online Learning via Cooperative Contextual Bandits

arXiv.org Machine Learning

In this paper we propose a novel framework for decentralized, online learning by many learners. At each moment of time, an instance characterized by a certain context may arrive to each learner; based on the context, the learner can select one of its own actions (which gives a reward and provides information) or request assistance from another learner. In the latter case, the requester pays a cost and receives the reward but the provider learns the information. In our framework, learners are modeled as cooperative contextual bandits. Each learner seeks to maximize the expected reward from its arrivals, which involves trading off the reward received from its own actions, the information learned from its own actions, the reward received from the actions requested of others and the cost paid for these actions - taking into account what it has learned about the value of assistance from each other learner. We develop distributed online learning algorithms and provide analytic bounds to compare the efficiency of these with algorithms with the complete knowledge (oracle) benchmark (in which the expected reward of every action in every context is known by every learner). Our estimates show that regret - the loss incurred by the algorithm - is sublinear in time. Our theoretical framework can be used in many practical applications including Big Data mining, event detection in surveillance sensor networks and distributed online recommendation systems.


Truth Serums for Massively Crowdsourced Evaluation Tasks

arXiv.org Artificial Intelligence

A major challenge in crowdsourcing evaluation tasks like labeling objects, grading assignments in online courses, etc., is that of eliciting truthful responses from agents in the absence of verifiability. In this paper, we propose new reward mechanisms for such settings that, unlike many previously studied mechanisms, impose minimal assumptions on the structure and knowledge of the underlying generating model, can account for heterogeneity in the agents' abilities, require no extraneous elicitation from them, and furthermore allow their beliefs to be (almost) arbitrary. These mechanisms have the simple and intuitive structure of an output agreement mechanism: an agent gets a reward if her evaluation matches that of her peer, but unlike the classic output agreement mechanism, this reward is not the same across evaluations, but is inversely proportional to an appropriately defined popularity index of each evaluation. The popularity indices are computed by leveraging the existence of a large number of similar tasks, which is a typical characteristic of these settings. Experiments performed on MTurk workers demonstrate higher efficacy (with a $p$-value of $0.02$) of these mechanisms in inducing truthful behavior compared to the state of the art.


The Siri of the cell – tech podcast

The Guardian

How can scientists deal with the huge volume of new research publish on a daily basis? How can computers go further than merely parsing scientific papers, and actually suggest hypotheses themselves? When will we see a computer as another member of the lab team, serving hundreds of scientists simultaneously from its huge data set of extant research? This is the work of John Bachman, a systems biology PhD from Harvard Medical School, and Ben Giori, a postdoctoral fellow at Harvard Medical School's systems pharmacology lab. They're part of Darpa's Big Mechanism project, which is developing technology to read research abstracts and papers to extract pieces of causal mechanisms, then to assemble these pieces into more complete causal models, and to produce explanations.


Best deals for Dec. 7: Instant Pot, LG OLED TVs, Samsung QLED TVs, Google Home, Facebook Portal

Mashable

That's how the expression goes, right? With the weekend now here, w've rounded up the best deals that should make it easier to rest and relax during your days off. We found deals on 4K smart TVs, kitchen products like Instant Pot and Vitamix, smart home gear from Google and Facebook, and Amazon devices for video streaming and home security. There are also deals on Udemy online courses for web development, animation, and more starting at $10.99. Save $131 on KitchenAid Deluxe 4.5 Quart Stand Mixer at Walmart Here are the rest of the best deals from across the internet for Friday, Dec. 7: Save on brilliant 4K TVs.