Adobe and Meta Decry Misuse of User Studies in Computer Vision Research

Adobe and Meta, together with the University of Washington, have published an extensive criticism regarding what they claim to be the growing misuse and abuse of user studies in computer vision (CV) research. User studies were once typically limited to locals or students around the campus of one or more of the participating academic institutions, but have since migrated almost wholesale to online crowdsourcing platforms such as Amazon Mechanical Turk (AMT). Among a wide gamut of grievances, the new paper contends that research projects are being pressured to produce studies by paper reviewers; are often formulating the studies badly; are commissioning studies where the logic of the project doesn't support this approach; and are often'gamed' by cynical crowdworkers who'figure out' the desired answers instead of really thinking about the problem. The fifteen-page treatise (titled Towards Better User Studies in Computer Graphics and Vision) that comprises the central body of the new paper levels many other criticisms at the way that crowdsourced user studies may actually be impeding the advance of computer vision sub-sectors, such as image recognition and image synthesis. Though the paper addresses a much broader tranche of issues related to user studies, its strongest barbs are reserved for the way that output evaluation in user studies (i.e. when crowdsourced humans are paid in user studies to make value judgements on – for instance – the output of new image synthesis algorithms) may be negatively affecting the entire sector.