We introduce a novel cyclic Markov decision process (MDP) framework for multi-step decision problems with heterogeneous stage-specific dynamics, transitions, and discount factors across the cycle. In this setting, offline learning is challenging: optimizing a policy at any stage shifts the state distributions of subsequent stages, propagating mismatch across the cycle. To address this, we propose a modular structural framework that decomposes the cyclic process into stage-wise sub-problems. While generally applicable, we instantiate this principle as CycleFQI, an extension of fitted Q-iteration enabling theoretical analysis and interpretation. It uses a vector of stage-specific Q-functions, tailored to each stage, to capture within-stage sequences and transitions between stages. This modular design enables partial control, allowing some stages to be optimized while others follow predefined policies. We establish finite-sample suboptimality error bounds and derive global convergence rates under Besov regularity, demonstrating that CycleFQI mitigates the curse of dimensionality compared to monolithic baselines. Additionally, we propose a sieve-based method for asymptotic inference of optimal policy values under a margin condition. Experiments on simulated and real-world Type 1 Diabetes data sets demonstrate CycleFQI's effectiveness.

We study offline reinforcement learning under $Q^\star$-approximation and partial coverage, a setting that motivates practical algorithms such as Conservative $Q$-Learning (CQL; Kumar et al., 2020) but has received limited theoretical attention. Our work is inspired by the following open question: "Are $Q^\star$-realizability and Bellman completeness sufficient for sample-efficient offline RL under partial coverage?" We answer in the negative by establishing an information-theoretic lower bound. Going substantially beyond this, we introduce a general framework that characterizes the intrinsic complexity of a given $Q^\star$ function class, inspired by model-free decision-estimation coefficients (DEC) for online RL (Foster et al., 2023b; Liu et al., 2025b). This complexity recovers and improves the quantities underlying the guarantees of Chen and Jiang (2022) and Uehara et al. (2023), and extends to broader settings. Our decision-estimation decomposition can be combined with a wide range of $Q^\star$ estimation procedures, modularizing and generalizing existing approaches. Beyond the general framework, we make further contributions: By developing a novel second-order performance difference lemma, we obtain the first $ε^{-2}$ sample complexity under partial coverage for soft $Q$-learning, improving the $ε^{-4}$ bound of Uehara et al. (2023). We remove Chen and Jiang's (2022) need for additional online interaction when the value gap of $Q^\star$ is unknown. We also give the first characterization of offline learnability for general low-Bellman-rank MDPs without Bellman completeness (Jiang et al., 2017; Du et al., 2021; Jin et al., 2021), a canonical setting in online RL that remains unexplored in offline RL except for special cases. Finally, we provide the first analysis for CQL under $Q^\star$-realizability and Bellman completeness beyond the tabular case.

Furthermore, whencontrollingcomplex high-dimensional morphologies, such as humanoid bodies, tasks often require coordination of multiple skills simultaneously.

OfflineRLisdeemed to be promising [16, 14] as online learning requires the agent to continuously interact with the environment, which howevermaybecostly,time-consuming, orevendangerous.

Deep RL(DRL) tackles the curse of dimensionality due to large state spaces by utilizing a Deep Neural Network (DNN) to approximate the value function and/or the policy.

Neural Information Processing Systems http://nips.cc/

This approach is often preferable when using "standard" parametric5 regression algorithms, as the weighted p-norm can be directly minimized at learning time (e.g., in least-squares6 regression,the`2-normisminimized). This is not surprising as in the worst case the inherent Bellman error may12 be unbounded and standard AVI tends to diverge. Recent work (Jinglin Chen, Nan Jiang,Information-Theoretic13 Considerations in Batch Reinforcement Learning, ICML 2019, Conjecture 8) has even conjectured an exponential14 lower bound in case of unbounded Bellman error. In the paper we propose afirst heuristic algorithm to automatically construct aset22 ofanchor points (see beginning ofpage 6). Akeybenefit ofourapproach isthat itdoes notmodify theunderlying (linear) feature51 representation, allowing theuser tousethelinear representation with, forexample, approximate value iteration, and52 should this fail, the user can switch toour algorithm and progressively increase the number of support points while53 keeping the same feature representation.

Delusionalbias ariseswhenthe approximation architecture limits the class ofexpressible greedy policies.