Sample efficiencyof ESRL is independent of the chosen risk aversion threshold and quality of the behavior policy.

Offline reinforcement learning (RL) tasks require the agent to learn from a precollected dataset with no further interactions with the environment. Despite the potential tosurpass thebehavioral policies, RL-based methods aregenerally impractical duetothetraining instability andbootstrapping theextrapolation errors, which always require careful hyperparameter tuning via online evaluation.

Offline meta-reinforcement learning (OMRL) utilizes pre-collected offline datasets to enhance the agent's generalization ability on unseen tasks.

We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data.

Based on this, we propose a novel approach, Diffusion-DICE, that directly performs this transformation using diffusion models.