We revisit the finite-armed linear bandit model by Nelson et al. (2022), where contexts and rewards are governed by a finite hidden Markov chain. Nelson et al. (2022) approach this model by a reduction to linear contextual bandits; but to do so, they actually introduce a simplification in which rewards are linear functions of the posterior probabilities over the hidden states given the observed contexts, rather than functions of the hidden states themselves. Their analysis (but not their algorithm) also does not take into account the estimation of the HMM parameters, and only tackles expected, not high-probability, bounds, which suffer in addition from unnecessary complex dependencies on the model (like reward gaps). We instead study the more natural model incorporating direct dependencies in the hidden states (on top of dependencies on the observed contexts, as is natural for contextual bandits) and also obtain stronger, high-probability, regret bounds for a fully adaptive strategy that estimates HMM parameters online. These bounds do not depend on the reward functions and only depend on the model through the estimation of the HMM parameters.

Various theoretical results provideupper bounds onthedistance between the target and marginal distribution after a fixed number of iterations.

Howeverwhat is perhaps more surprising, is that in stark contrast to our classic understanding of generalization in machine learning models, this does not seem to degrade the generalization capabilities of the learned model in spite of the significant increase in its capacity.

However, vanilla Adam remains exceptionally popular anditworkswellinpractice.

Our bounds hold ininfinite-dimensional spaces, thereby showing that finer and finer discretizations do not make this learning problemharder.

Inboth cases, weestablish that the number of calls to the stochastic first-order oracle to obtain an appropriately definedϵ-stationary point isoftheorderO(1/ϵ2.5).

During the exploration phase, an agent collects samples without using a pre-specified reward function. After the exploration phase, a reward function is given, and the agent uses samples collected during the exploration phase to computeanear-optimalpolicy.

One of the fundamental aspects of over-parametrized models is that they are capable of interpolating the training data. We show that, under interpolation-like assumptions satisfied by the stochastic gradients in an overparametrization setting, thefirst-order oracle complexityofPerturbed Stochastic Gradient Descent (PSGD) algorithm toreach an -local-minimizer,matches the corresponding deterministic rateof O(1/2).

The above problem requires the knowledge of the distributionπ which is typically unknown in practice.