Appendix T able of Contents

The actor losses used in DoubleGum, SAC, and DDPG are all derived from the same principle. SAC (Haarnoja et al., 2018a,b) has a policy with learned variance and state-independent Section B.1 shows this for the actor losses of DoubleGum, SAC, and DDPG. We now relate the critic losses to each other, starting from the most general case, DoubleGum. The SAC noise model is derived from Equation 16 in three ways. In continuous control, Fujimoto et al. (2018) introduced Twin Networks, a method that improved Follow-up work selects a quantile estimate from an ensemble (Kuznetsov et al., 2020; Chen et al., 2021; Ball et al., 2023), which we demonstrate is Moskovitz et al. (2021) and Ball et al. (2023) showed that the appropriate Garg et al. (2023) present a method of estimating its value using Gumbel regression.