Parameters5 10 3 5 10 3 5 10 3 (Critic Update:Environment Step) Ratio 1 1 1 (Policy Update:Environment Step) Ratio 2 2 1 Has Target Policy?

While this could be done using thesynthesis formulation, we demonstrate that this leads to slower performances. The main difficulty inapplying suchmethods intheanalysisformulation liesinproposing a way to compute the derivatives through the proximal operator.

Then, we propose a meta-learning framework, to jointly train the perturbation function over heterogeneous tasks in parallel. As MetaPerturb is a set-function trained over diverse distributions across layers and tasks, it can generalize toheterogeneous tasksandarchitectures.

We propose a non-iterative method that enforces the following ideas during training.

Insuch situations, weallowthealgorithm toreturn asmall listofcandidates such that one of the candidates is close to the true parameter.