Contents of Appendix A Extended Literature Review 14 B Time Uniform Lasso Analysis 15 C Results on Exploration 18 C.1 ALE 20 C.2 Proof of Results on Exploration 20 D Proof of Regret Bound