Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs

Large Language Models (LLMs) have demonstrated remarkable progress in understanding and generating human language, scaling from initial architectures capable of handling a few hundred tokens to recent systems supporting context windows exceeding 100,000 tokens. Such expansions in context length enable LLMs to process extensive documents--including entire novels, legal contracts, or scientific reports--in a single prompt. Despite this progress, LLMs still face critical challenges when dealing with very long inputs containing dispersed, multi-faceted information (Kuratov et al., 2024; Li et al., 2024; Wang et al., 2024). In particular, simply increasing context size does not guarantee that a model will accurately retrieve and combine distant pieces of information. Models often fail on tasks requiring multi-hop reasoning, especially when relevant details are scattered across large portions of text (Lee et al., 2024; Adams et al., 2024; Karpinska et al., 2024; Levy et al., 2024). One promising line of research has focused on Retrieval-Augmented Generation (RAG), in which an LLM is augmented with an external retrieval mechanism to fetch relevant passages from a massive corpus or knowledge base. This approach efficiently narrows down the input, enabling the model to focus on a set of shorter, contextually relevant documents (Reddy et al., 2024). While RAG excels at pinpointing factual snippets in long contexts, it often struggles with multi-hop or compositional reasoning: if multiple evidence fragments must be pieced together, a naive retrieve-then-summarize workflow can fail to integrate that information cohesively (Bai et al., 2024). In parallel, Chain-of-Thought (CoT) prompting has emerged as a powerful technique for guiding LLMs through explicit intermediate reasoning steps (Wei et al., 2022).