As an important framework for safe Reinforcement Learning, the Constrained Markov Decision Process (CMDP) has been extensively studied in the recent literature. However, despite the rich results under various on-policy learning settings, there still lacks some essential understanding of the offline CMDP problems, in terms of both the algorithm design and the information theoretic sample complexity lower bound. In this paper, we focus on solving the CMDP problems where only offline data are available.

In this paper we develop a new reparameterization-based methodology that allows for generating programs whose expectation is the derivative of the expectation of the original program.

FCL T results in the Markov chain setting, an alternative condition that is more applicable for SA problems is established.

Continuous hidden Markov models (HMMs) assume that observations are generated from a mixture of Gaussian densities, limiting their ability to model more complex distributions.

We will first replicate an equation similar to (20) for the backward case. The derivation is similar to that of the forward equation, so that it uses a combination of equations (16), (18) and (19) while leaving out the observation likelihood function. The combination is again carried out using the Laplace transform.

Hidden semi-Markov Models (HSMM's) - while broadly in use - are restricted to

One of the challenges in online reinforcement learning (RL) is that the agent needs to trade off the exploration of the environment and the exploitation of the samples to optimize its behavior. Whether we optimize for regret, sample complexity, state-space coverage or model estimation, we need to strike a different exploration-exploitation trade-off.