Appendix A Source codes

Source codes for reproducing our experimental results are available at https://github.com/ To encourage the size of the dataset to be consistent across multiple environments, we use the number of expert demonstrations N 2{20, 50}. We provide the size of a dataset for each environment in Table 4. Following de Haan et al. [12], we consider confounded Atari environments, where images are augmented with previous actions (see Figure 4). We provide source codes for loading images from the dataset, preprocessing images, and augmenting numbers to the images in Section A. For experiments with selected environments in Figure 7, we randomly chose 8 confounded Atari environments, i.e., BankHeist, Enduro, KungFuMaster, Pong, PrivateEye, RoadRunner, Seaquest, and UpNDown, due to the high computational cost of considering all environments.