This is only for the ease of visualization. For linear MDP,In the generative model setting, Agarwal et al. [2020] shows model-based approach is still minimax optimal O((1 γ) 3SA/2)byusing as-absorbing MDP construction andthismodelbased technique is later reused for other more general settings (e.g. Itrequires high probability guarantee for learning optimal policyforany reward function, which is strictly stronger than the standard learning task that one only needs to learn to optimal policy for a fixed reward. B.2 GeneralabsorbingMDP The general absorbing MDP is defined as follows: for a fixed states and a sequence {ut}Ht=1, MDPMs,{ut}Ht=1 is identical toM for all states excepts, and state s is absorbing in the sense PMs,{ut}Ht=1(s|s,a) = 1 for all a, and the instantaneous reward at timet is rt(s,a) = ut for all a A. Also,weusetheshorthand notationVπ{s,ut} forVπs,Ms,{u We focus on the first claim. Later we shall remove the conditional onN (see SectionB.7). We use the singleton-absorbing MDPMs,{u?t}Ht=1 to handle the case (recallu?t

It is known that Binary Segmentation is consistent but not optimal (Venkatraman (1992)). As an improvement, Fryzlewicz (2014) propose WildBinary Segmentation andshowthatithasabetter localization rate.

In this work, we show for the first time that error compensated gradient compression methods can be accelerated. In particular, we propose and study the error compensated loopless Katyusha method, and establish an accelerated linear convergence rate under standard assumptions.

In the deterministic setting where the data is deterministically given without any probabilistic assumptions, significant advances inDP linear regression has been made [77,57,68, 16, 7, 83, 31, 67, 82, 71]. In the randomized settings where each example{xi,yi} is drawn i.i.d. We explain the closely related ones in Section 2.3, with analysis when the covariance matrixhasaspectralgap. The resulting utility guarantees are the same as those from [23], which are discussedinSection2.3. When privacy is not required, we know from Theorem 2.2 that under Assumptions A.1-A.3, we can achieve an error rate of O(κ p V/n).

We need some more notation in order to linearize the value function.

Consideranews recommendation website that, when presented with a new user, sequentially offers a selection of currently trending articles. Such asystem may only haveafewopportunities tomakerecommendations before the user decides to navigate away, leaving little time to correct for misspecified or underspecified prior knowledge.

From the Posterior Sampling Lemma, we know that ifψ is the distribution off, then for any sigma-algebraσ(Ht)-measurablefunctiong, E[g(f)|Ht]=E[g(ft)|Ht]. We can further know from the construction of the confidence set (c.f. This lemma is widely adopted in RL. Proof can be found in various previous works, e.g. Prior work that shares similarities with ours contains DPI [59]and GPS [31,39]as dual policyoptimization procedures areadopted.

We show that, surprisingly, the notion of optimal finite-time regret is not auniquely defined property in this context and that, in general, it is decoupled from theasymptotic rate.

In this setting, unlike basic setting, objective and constraints are not linear. We focus on a single state-action pairs,a, stage h, and objectivem. Similarly, in constrained settings, its estimated resource consumptions are underestimates of the true resource consumptions. B.5 BoundingtheBellmanerror We now provide an upper bound on the Bellman error which arises in the RHS of the regret decomposition(Proposition3.3). When neither failure events occur (probability 1 2δ), Proposition 3.3 upper bounds either of reward or consumption regret by In this section, we prove the main guarantee for the convex-concave setting.

First,theobtained high-probability regret bounds are data-dependent and could be much smaller thantheworst-case bounds, which resolvesanopenproblem askedbyNeu[31].