Instead, we have access to a dataset generated from some past suboptimal policies.

The performances of the agents are in line with the results reported in Osband et al.

Finally, we benchmark five tabular agents in our newly proposed benchmark.

While this is a common approach in supervised learning, to our knowledge, this has not been discussed in detail in the offline RL setting.

This problem has been studied in stochastic optimal control, particularly REPS [Peters et al., 2010]. In our experiments, we use soft actor-critic [Haarnoja et al., 2018] as our base RL algorithm. The policy and critic networks are MLPs with 2 fully-connected hidden layers of size 256. Following [Sharma et al., 2021b], we use a biased TD update, where For all experiments using prior data collected through RL, the agent was initialized at test time with the pretrained policy and critic. The details for this environment are in [Sharma et al., 2021b].

For example, imagine a disaster relief robot tasked with retrieving an item from a fallen building, where it cannot get direct supervision from humans. It must retrieve this object within one test-time trial, and must do so while tackling unknown obstacles, though it may leverage knowledge it has of the building before the disaster.

In contrast, reinforcement learning requires human-designed rewards to accomplish any desired outcome.

In this work, we focus on the setting where unsafe states can be avoided by planning ahead a short time into the future. In this setting, a model-based agent with a sufficiently accurate model can avoid unsafe states.