Consensus measure of rankings

In many information systems, rankings are widely used to represent the preferences over a set of items or candidates, ranging from information retrieval, recommender to decision making systems [1], [2], [3], [4], [5], [6], in order to improve quality of the services provided by the systems. For example, in search engine, the list of the terms suggested by a search engine after a user's few keystrokes is a typical ranking and such ranking service, widely adopted nowadays, has great impact on user's search experience; it is also recognized that the list of search results is a ranking after a query is issued. A ranking is an ordered sequence of items, in which an item with higher ranking score is more preferred than the items with lower ranking scores. The consensus of rankings is the degree to which the rankings agree according to certain common patterns. The consensus measure, can be used in many information systems, in order to uncover how close or related the rankings are. For example, in the group decision making, a group of experts express their preferences over a set of candidates by using rankings and the measure of the degree of consensus is very useful for reaching consensus [2]. In many information system with large volume of items, such as search engines, it is hard to clearly define what ground truth is, which make it more difficult to evaluate and compare the rankings returned from the systems. The consensus measure of rankings, as a tool for understanding how related or close the rankings are, will help engineers and researchers to discern what aspects of a ranking system need to be improved and to detect outliers [7], [8].