Conformal Ranked Retrieval

Ranked retrieval refers to the process of retrieving and ranking documents from a document repository based on their relevance to a user's query. As the core component in Information Retrieval (IR) systems, its goal is to present the most relevant documents at the top of the search results list, making it easier for users to find the information they seek (Baeza-Yates and Ribeiro-Neto, 1999). Over the years, ranked retrieval techniques have been successfully applied to many real-life problems, including web search engines, recommendation systems, and question-and-answer platforms, significantly impacting our daily lives. While ranked retrieval algorithms have been extensively studied in both academia and industry, considering the uncertainty in their predictions is a relatively new challenge. As we increasingly rely on search engines for answers to a wide variety of questions, it becomes crucial to evaluate the reliability of these retrieved answers. Therefore, it is important to quantify the uncertainty of the results, determining whether they encompass all the desired documents and whether these documents are ranked in a reasonable order. The challenges, however, lie in measuring uncertainty for ranked retrieval algorithms and developing methodologies to control this uncertainty. This is particularly challenging due to the complexity of ranked retrieval systems, which typically consist of multiple stages, each with different optimization goals.