Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments

Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents D ocument R etrieval-A ugmented F ine-T uning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) fo r safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by intro ducing a novel fine-tuning framework that accommodates our dual-re trieval architecture, which simultaneously accesses both softwar e documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodolog y that incorporates variable numbers of relevant documents with m eaning-ful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT -4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative impr ovements in evidence handling, response structure, and domain-spec ific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transpar ency and evidence-based reasoning essential in regulatory domains .