A version ofthis bound becomes tight under certain assumptions.

In this work, we identify anovel set of conditions that ensure convergence with probability 1 ofQ-learning with linear function approximation, by proposing a twotime-scalevariationthereof.

However, achieving such an assumption is non-trivial, leading to the following two aspects of challenges. On the one hand, it is challenging to design a model with enough capacity with limited computational resources, andexisting models areusually tailored forspecific problems which require heavytrailand-error [25,57,59].