A Additional Implementation Details

These hyperparameters are fixed throughout all domains. Tab. 1 details the hyper-parameters used in MOSS which are taken directly from We include the environment renders in Figure?? . 1 Table 2: Hyperparameters for MOSS and DQN. These hyperparameters are fixed throughout all domains. Action repeat 1 Frame repeat 12 Seed frames 4000 n-step returns 3 Mini-batch size 1048 Discount ( γ) 0.99 Optimizer Adam Learning rate 0.0001 Agent update frequency 2 Critic target EMA rate ( τ We made modifications to MOSS to evaluate in discrete action settings. Tab. 2 details the hyper-parameters used for Double DQN and MOSS in the ViZDoom environment.