We propose WSAC (Weighted Safe Actor-Critic), a novel algorithm for Safe Offline Reinforcement Learning (RL) under functional approximation, which can robustly optimize policies to improve upon an arbitrary reference policy with limited data coverage. WSAC is designed as a two-player Stackelberg game to optimize a refined objective function. The actor optimizes the policy against two adversarially trained value critics with small importance-weighted Bellman errors, which focus on scenarios where the actor's performance is inferior to the reference policy. In theory, we demonstrate that when the actor employs a no-regret optimization oracle, WSAC achieves a number of guarantees: (i) For the first time in the safe offline RL setting, we establish that WSAC can produce a policy that outperforms {\bf any} reference policy while maintaining the same level of safety, which is critical to designing a safe algorithm for offline RL. (ii) WSAC achieves the optimal statistical convergence rate of 1/\sqrt{N} to the reference policy, where N is the size of the offline dataset. Additionally, we offer a practical version of WSAC and compare it with existing state-of-the-art safe offline RL algorithms in several continuous control environments.

Many safe RL approaches have been proposed in the past few years (Achiam et al., Aiming at promoting the safe real-world deployment 2017; Zhang et al., 2020; Sootla et al., 2022; Liu et al., of Reinforcement Learning (RL), research 2022a). Unfortunately, most existing approaches only target on safe RL has made significant progress in recent at the online setting, where potentially risky constraint years. However, most existing works in the violations can be incurred during interactions with the real literature still focus on the online setting where environment. As a kind of data-driven methods, offline risky violations of the safety budget are likely to RL (Levine et al., 2020) aims to derive a policy from offline be incurred during training. Besides, in many realworld data without further real-world exploration, and thus is particularly applications, the learned policy is required suitable for safety-critical applications. Despite the to respond to dynamically determined safety budgets recent progress in the offline RL literature (Fujimoto et al., (i.e., constraint threshold) in real time. In 2019; Kumar et al., 2020; Fujimoto & Gu, 2021), however, this paper, we target at the above real-time budget there are still limited works focusing on attaining a safe constraint problem under the offline setting, policy under the offline setting.