While knowntobesample efficient, these methods havefailed tofully leverage recent advances indeep learning, forcing the use of less efficient but more scalable model-free methods which try to learn the values directly.

Ateachh2[H] inanepisodek, thealgorithmsh, k, selects ah, k h, k(sh, k, ), and costsrh(sh, k,ah, k)andch(sh, k,ah, k). Wewillalsoshowthat k from (10) (onceitbecomes feasible) willindeedbeasafepolicy (see Proposition 5).

However, instead of gradients, the critic is, typically, only trained to accurately predict expected returns, which, on their own, are useless for policy optimization.

A wide array of control applications, ranging from medical to engineering, fundamentally deals with critical systems, i.e., systems of vital importance where the control actions have to guarantee no harm to the system functionality. Examples include managing nuclear fusion [Degrave et al., 2022], performing robotic surgeries [Datta et al., 2021], and devising patient treatment strategies [Komorowski et al., 2018].

High sample complexity has long been a challenge for RL.