In this section, we describe details about hyperparameter setting of RD. SAC-N-Unc and TD3-N-Unc, M is set to 1/10 of the total training steps. To ensure fairness, algorithms employing RD are implemented using CORL repository [54]. By modifying the original SAC/TD3 algorithm to employ a critic ensemble of number N and incorporate an uncertainty regularization term within the policy update process, we derive these backbone algorithms. Additionally, using RD with fewer Q ensembles can achieve similar or even better results than the backbone methods using more Q ensembles, indicating its potential in reducing computing resource consumption.

Note that ˆφ is feasible for the constrained optimization problem. We refer to it as an "early stopping scheme" because the key idea is to return to the parameter values which gave the lowest validation error (see Section 7.8 of Goodfellow et al.[3]). In our implementation, we initialize two upper envelope networks with parametersφ and φ0, where φ is trained using the penalty loss, andφ0 records the parameters with the lowest validation error encounteredsofar. IfLφ > Lφ0, we count the number of consecutive times this occurs. Notonlyis this not standard practice, but to makeafair comparison across all algorithms, this would require, foreachofthe fivealgorithms, performing aseparate hyper-parameter search foreachofthe five environments.

Once estimates, viagradient error: = r (Q (s, a) target) 2. Ourfocus Q-networks 2 Where Do Learned Fourier Features Help?

Imitation learning (IL) aims to learn a policy from expert demonstrations that minimizes the discrepancy between the learner and expert behaviors.

In economics, Return on Investment (ROI) is a financial metric used to evaluate the profitability of an investment relative to its cost. The concept of ROI originates from the work of [5] and is widelyregardedasavaluable metricbythemajority ofmarketingmanagers [4].