ColPali: Efficient Document Retrieval with Vision Language Models

Documents are visually rich structures that convey information through text, as well as tables, figures, page layouts, or fonts. While modern document retrieval systems exhibit strong performance on query-to-text matching, they struggle to exploit visual cues efficiently, hindering their performance on practical document retrieval applications such as Retrieval Augmented Generation. To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieving tasks spanning multiple domains, languages, and settings. The inherent shortcomings Figure 1: For each term in a user query, ColPali identifies of modern systems motivate the introduction the most relevant document image patches (highlighted of a new retrieval model architecture, zones) and computes a query-to-page matching ColPali, which leverages the document score. We can then swiftly retrieve the most relevant understanding capabilities of recent Vision Language documents from a large pre-indexed corpus. Models to produce high-quality contextualized embeddings solely from images of document pages.