In order to overcome overestimation bias, ensemble methods for Q-learning have been investigated to exploit the diversity of multiple Q-functions. Since network initialization has been the predominant approach to promote diversity in Q-functions, heuristically designed diversity injection methods have been studied in the literature. However, previous studies have not attempted to approach guaranteed independence over an ensemble from a theoretical perspective.

We use induction to prove this statement. The penultimate step follows from the induction hypothesis completing the proof. Then, the fixed point of Eq.(5) is the value function of in f M . We focus on permanent value function in the next two theorems. The permanent value function is updated using Eq.

"subagents," one for each subproblem, and then combine their solutions to establish

Neural Information Processing Systems http://nips.cc/

Weproveconvergence for this method under standard assumptions and demonstrate empirically that it indeed enables lowerdiscount factors forapproximate reinforcement-learning methods.

Delusionalbias ariseswhenthe approximation architecture limits the class ofexpressible greedy policies.