Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models

Relatedness-based evaluation - known as intrinsic evaluation in the context of embedding-based A standard approach to evaluate a relatednessbased models - requires the construction of a dataset of model is the comparison of the semantic human annotations, which may be collected via ranking it produces with the corresponding ranking two different approaches. The former relies on a determined from human annotations. However, small group of linguistic experts to create a gold the relevance of rank mismatches may depend standard dataset, which is reliable but very expensive on the involved positions; in particular, top ranks and, due to the subjectivity of relatedness and are considered more important in many contexts, to the limited number of annotations, highly susceptible two prominent examples being content-based recommenders to bias and lack of statistical significance (De Gemmis et al., 2008, 2015; Lops (Blanco et al., 2013; Faruqui et al., 2016). The latter et al., 2011; Mladenic, 1999) and semantic matching relies on a large group of non-experts, typically (Giunchiglia et al., 2004; Li and Xu, 2014; associated with a crowdsourcing service (e.g., Amazon Wan et al., 2016). The greater significance of top MTurk, ProlificAcademic, SocialSci, Crowd-ranks compared with low ranks is actually a pretty Flower, ClickWorker, CrowdSource), it is typically common phenomenon, as it can be argued from more affordable, and it has been proven to be repeatable the attempts to overweight the former in the context and reliable (Blanco et al., 2013). of ranking correlation (Blest, 2000; Pinto da In the next sections we describe and justify a Costa and Soares, 2005; Dancelli et al., 2013; Iman protocol to construct a dataset based on semantic and Conover, 1987; Maturi and Abdelfattah, 2008; relatedness between pairs of tokens