Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity

A fundamental challenge of deploying RL to real-world applications is the limited ability to explore or interact with the environment, due to resources, time, or safety constraints. Offline RL, or batch RL, seeks to circumvent this challenge by resorting to history data--which are often collected by executing some possibly unknown behavior policy in the past--with the hope that the history data might already provide significant insights about the targeted optimal policy without further exploration (Levine et al., 2020). Besides maximizing the expected total reward, perhaps an equally important goal--to say the least--for an RL agent is safety and robustness (Garcıa and Fernández, 2015), especially in high-stake applications such as robotics, autonomous driving, clinical trials, financial investments, and so on (Choi et al., 2009; Schulman et al., 2013). It has been observed that a standard RL agent trained in an ideal environment might be extremely sensitive and fail catastrophically when the deployed environment is subject to small adversarial perturbations (Zhang et al., 2020). Consequently, robust RL has attracted a surge of attentions with the goal to learn an optimal policy that is robust to environment perturbations. In fact, providing robustness guarantees becomes even more relevant in the offline setting, which can be formulated as robust offline RL, since the history data is often inevitably collected from a timeframe where it is no longer reasonable to assume model stillness, due to the highly non-stationary and time-varying dynamics of many real-world applications. Altogether, this naturally leads to a question: Can we learn a near-optimal policy which is robust with respect to uncertainties and variabilities of the environments using as few history samples as possible?