The training of language models is typically not an end-to-end process.

A key metric to determine which gradient estimator is effective is thevariance of the gradient estimator,V[ˆg].

InIRsystems, theretrievermodule directly influences theperformance ofdownstream tasks, such as retrieval-augmented generation [20, 29, 30] and knowledge-intensive question answering [34, 52]. However, these methods do not explicitly consider targeted optimization for tool usage or the impact on complex multi-stage tasks.

A contiguous chunk of lines to search for in the existing sourcecode 4. The dividing line: =======5. The lines to replace into the source code6. The end of the replace block: >>>>>>> REPLACEHere is an example: '''python ### mathweb/flask/app.py<<<<<<< SEARCH from flask import Flask ======= import math from flask import Flask >>>>>>> REPLACE ''' Please note that the * SEARCH/REPLACE * edit REQUIRES PROPER INDENTATION.If you would like to add the line ' print(x)', you mustfully write that out, with all those spaces before the code!Wrap the * SEARCH/REPLACE * edit in blocks '''python...'''.The summary of the key differences between the trajectories should bein the thinking part.

Large Language Models (LLMs) have been widely adopted as task planners for AI agents in sequential decision-making problems, leveraging their extensive world knowledge. However, the gap between their general knowledge and environment-specific requirements often leads to inaccurate plans. To address this, existing approaches rely on frequent LLM queries to iteratively refine plans based on immediate environmental feedback, which incurs substantial query costs. However, this refinement is typically guided by short-term environmental feedback, limiting LLMs from developing plans aligned with long-term rewards. We propose Code Driven Planning with Domain-Adaptive Critic (CoPiC). Instead of relying on frequent queries, CoPiC employs LLMs to generate a diverse set of high-level planning programs, which iteratively produce and refine candidate plans. A trained domain-adaptive critic then evaluates these candidates and selects the one most aligned with long-term rewards for execution. Using high-level planning programs as planner and domain-adaptive critic as estimator, CoPiC improves planning while significantly reducing query costs. Results in ALFWorld, NetHack, and StarCraft II Unit Building show that CoPiC outperforms advanced LLM-based baselines, AdaPlanner and Reflexion, achieving an average (1) 23.33% improvement in success rate and (2) 91.27% reduction in query costs.