Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in realworld environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios. Offline reinforcement learning (RL) is an increasingly prominent paradigm for learning decisionmaking from offline datasets (Levine et al., 2020; Fujimoto et al., 2019; Kumar et al., 2019; 2020). Its primary objective is to address the limitations of RL, which traditionally necessitates a large amount of online interactions with the environment. Despite its advantages, offline RL suffers from the challenges of distribution shift between the learned policy and the data distribution (Levine et al., 2020). Most previous studies on offline RL have primarily focused on simulation tasks, where data collection is relatively accurate. However, in real-world environments, data collected by humans or sensors can be subject to random noise or even malicious corruption. For example, during RLHF data collection, annotators may inadvertently or deliberately provide incorrect responses or assign higher rewards for harmful responses.