A version ofthis bound becomes tight under certain assumptions.

In this work, we identify anovel set of conditions that ensure convergence with probability 1 ofQ-learning with linear function approximation, by proposing a twotime-scalevariationthereof.

However, achieving such an assumption is non-trivial, leading to the following two aspects of challenges. On the one hand, it is challenging to design a model with enough capacity with limited computational resources, andexisting models areusually tailored forspecific problems which require heavytrailand-error [25,57,59].

Itisknown that policyevaluation has the interpretation of solving ageneralized Bellman equation. Inthispaper,wederivefinite-sample bounds foranygeneral off-policy TD-like stochastic approximation algorithm that solves for the fixedpoint of this generalized Bellman operator.

In this paper, we focus on testing for safety.