Re-ranking the Context for Multimodal Retrieval Augmented Generation

Abstract--Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge to generate a response within a context with improved accuracy and reduced hallucinations. However, multi-modal RAG systems face unique challenges: (i) the retrieval process may select irrelevant entries to user query (e.g., images, documents), and (ii) vision-language models or multi-modal language models like GPT-4o may hallucinate when processing these entries to In this paper, we aim to address the first challenge, i.e, improving the selection of relevant context RAG. Specifically, we leverage the relevancy score (RS) measure designed in our previous work for evaluating the RAG performance The work in [4] large language models (LLMs) [2] by incorporating external employs multi-modal large language models (MLLMs) with knowledge to generate coherent responds based on a given knowledge-enhanced re-ranking and noise-injected training to context, improve the response accuracy and reduce hallucinations. Also, the work in [6] demonstrates the However, the quality of the generated response in RAG potential of vision-language models for relevance evaluation, systems heavily depends on the retrieval process. Selecting but it highlights these models' tendency to over-rely on the most relevant data from a database based on the user semantic similarity, often failing in cases that require precise query is essential for the system to generate accurate and contextual understanding. A common approach for approaches is their reduced ability to detect irrelevant retrieval in RAG is top-k selection by first ranking the entries information, as MLLMs are primarily trained on datasets from the knowledge-base based on similarity scores between containing relevant image-query pairs.