A Network Architecture

For a fair comparison, our network follows the same structure as CEM-RL [19]. The architecture is originally from Fujimoto et al. [5], the only difference is using tanh instead of RELU. We use (400, 300) hidden layer for all environment except Humanoid-v2. For Humanoid-v2, we used (256, 256) as in TD3 [5]. Most of hyperparameters are the same value as CEM-RL [19].