We consider the reinforcement learning problem for the constrained Markov decision process (CMDP), which plays a central role in satisfying safety or resource constraints in sequential learning and decision-making.

However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient).

Non-convex optimization is a difficult and crucial task.

Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets.

Sparse ridge regression problems play a significant role across various domains.

Existing solutions either work under very specific assumptions or achieve bounds that are vacuous in some regimes.