Autonomous agents operating in real-world scenarios frequently encounter uncertainty and make decisions based on incomplete information.

Reward Machines provide an automaton-inspired structure for specifying instructions, safety constraints, and other temporally extended reward-worthy behaviour. By exposing the underlying structure of a reward function, they enable the decomposition of an RL task, leading to impressive gains in sample efficiency.

Sampling in discrete spaces, with critical applications in simulation and optimization, has recently been boosted by significant advances in gradient-based approaches that exploit modern accelerators like GPUs. However, two key challenges are hindering further advancement in research on discrete sampling.

Reconstructing past events requires reasoning across long time horizons.

This paper explores the realm of infinite horizon average reward Constrained Markov Decision Processes (CMDPs).