We note that the interchange of the integral and infinite summation is justified by Section 3.7 in [5], since the coefficients Z Now,define action sequence (a)n such thata1 = a and an = a1 for alln > 1. Then we can use subadditivity of measure to bound the maximum difference across all entries of [kZ]. Therefore, the induced infinity norm error ofbZ isless thanεifthe element wise error isless than ε/k. Therefore,bα>Fφ(s) is ρ-Lipschitz if the absolute value of its derivativeisboundedbyρ,i.e. SincebF has all zeros beyond thek-th column and row, each infinite-matrix bF can be treated as ak k matrix.

TheIRLsettingisremarkably useful for automated control, in situations where the reward function is difficult to specify manually or as a means to extract agent preference.

Moreover,our algorithm is model-free and provides a framework to justify the effectiveness of algorithms usedinpractice.

We achieve that by maintaining a replay buffer for each environment the agent interacts with.

Asmodern machinelearning systems become morepowerfulandembedded inourlives,theneed to have these systems explain their behavior becomes increasingly urgent.

Theorem 1.Let n 9. Underassumption(A2) with 1 < 1, thereexistapositivereal number independentofnsuchthat, for 0 , (a) Using = 0n Also, simplecomputationsshowthatV is anaffinetransformofr: V (x)= ar(x)+ b, witha =( 1 (1 ")) 1 andb = a Wealsoacknowledgesupport fromthe European Research Council (gran...