See Corollary 2 in Appendixfordetail.

Neural Information Processing Systems http://nips.cc/

Howevb isnon-zero Similarlyr(s, ) are features: r(s, a)= (s, a)>w. Assumption A3(Featureexcitation)Forapolicy withstationarydistributiond (s, a), define =E(s,a) d [ (s, a) (s, a)>].

Figure crepanc bythe which thee circles which differ .

Wemodel 34], whichisformallyG = hS, U, Pi. S is thestatespacet, every i 2 A {1,..., n} choosesui 2 U which action u 2 U Un. P(s0|s,u): S U S!

In with step-wise generally on-policy o with estimates model discrete a neural median Because trajectories T-step our IS/WIS T mo of20when re picks infinite map corresponding one T the iterations.

Neural Information Processing Systems http://nips.cc/

Next Token Distribution:Pn(w): $ ![0,1], defined:Pn(w)( )= Pl(w ).