RAG-VR: Leveraging Retrieval-Augmented Generation for 3D Question Answering in VR Environments

Figure 1: The hardware setup of our RAG-VR system including a VR device (left) and an edge server and a router (center), as well as an example of the user interface displayed to a VR user (right). Recent advances in large language models (LLMs) provide new opportunities for context understanding in virtual reality (VR). However, VR contexts are often highly localized and personalized, limiting the effectiveness of general-purpose LLMs. To address this challenge, we present RAG-VR, the first 3D question-answering system for VR that incorporates retrieval-augmented generation (RAG), which augments an LLM with external knowledge retrieved from a localized knowledge database to improve the answer quality. RAG-VR includes a pipeline for extracting comprehensive knowledge about virtual environments and user conditions for accurate answer generation. To ensure efficient retrieval, RAG-VR offloads the retrieval process to a nearby edge server and uses only essential information during retrieval. Moreover, we train the retriever to effectively distinguish among relevant, irrelevant, and hard-to-differentiate information in relation to questions. RAG-VR improves answer accuracy by 17.9%-41.8% As virtual reality (VR) continues to transform various facets of life, such as entertainment, social interactions, commerce, and education, there is a growing demand for VR applications endowed with context understanding capabilities [22, 19]. By gaining a detailed knowledge of virtual environments and VR users, these applications deliver immersive and personalized experiences, intelligently responding to user queries regarding their own conditions and surrounding 3D virtual objects.