Sample Complexity of Variance-reduced Distributionally Robust Q-learning

Reinforcement learning (RL) (Sutton and Barto, 2018) focuses on how agents can learn to make optimal decisions in an uncertain and dynamic environment. It is based on the idea of trial and error learning, where the agent learns by interacting with the environment, receiving rewards or penalties for its actions, and adjusting its behavior to maximize the expected long-term reward. Reinforcement learning faces a significant obstacle in the form of limited interaction between the agent and the environment, often due to factors such as data-collection cost or safety constraints. To overcome this, practitioners often rely on historical datasets or simulation environments to train the agent. However, this approach can suffer from distributional shifts (Quinonero-Candela et al., 2008) between the real-world environment and the data-collection/simulation environment, which can lead to a suboptimal learned policy when deployed in the actual environment. To tackle these challenges, distributionally robust reinforcement learning (DR-RL) (Zhou et al., 2021; Yang et al., 2021; Liu et al., 2022; Shi and Chi, 2022; Wang et al., 2023b) has emerged as a promising approach. DR-RL seeks to learn policies that are robust to distributional shifts in the environment by explicitly considering a family of possible distributions that the agent may encounter during deployment. This approach allows the agent to learn a policy that performs well across a range of environments, rather than just the one it was trained on.