Evaluating Retrieval Augmented Generative Models for Document Queries in Transportation Safety

Evaluating Retrieval A ugmented G enerative Models for Document Queries in Transportation Safety C.A. Melton, A. Sorokine, S. Peterson Oak Ridge National Laboratory, Oak Ridge, TN, United States National Security Sciences Directorate ABSTRACT Applications of generative Large Language Models (LLMs) are rapidly expanding across various domains, promising significant improvements in workflow efficiency and information retrieval. However, their implementation in specialized, high - stakes domains suc h as hazardous materials transportation is challenging due to accuracy and reliability concerns. This study evaluates the performance of three fine - tuned generative models -- ChatGPT, Google's Vertex AI, and ORNL Retrieval - Augmented Generation augmented LLaMA 2 and LLaMA in retrieving regulatory information essential for hazardous material transportation compliance in the United States. Utilizing approximately 40 publicly available federal and state regulatory documents, we developed 100 realistic queries relevant to route planning and permitting requirements. Responses were qualitatively rated based on accuracy, detail, and relevance, complemented by quantitative assessments of semantic similarity between model outputs. Results demon strated that the RAG - augmented LLaMA models significantly outperformed Vertex AI and ChatGPT, providing more detailed and generally accurate information, despite occasional inconsistencies. This research introduces the first known application of RAG in tra nsportation safety, emphasizing the need for domain - specific fine - tuning and rigorous evaluation methodologies to ensure reliability and minimize the risk of inaccuracies in high - stakes environments.