We initiate the mathematical study of replicability as an algorithmic property in the context of reinforcement learning (RL). We focus on the fundamental setting of discounted tabular MDPs with access to a generative model. Inspired by Impagliazzo et al. [2022], we say that an RL algorithm is replicable if, with high probability, it outputs the exact same policy after two executions on i.i.d. Next, for the subclass of deterministic algorithms, we provide a lower bound of order \Omega\left(\frac{N 3}{(1-\gamma) 3\cdot\varepsilon 2\cdot\rho 2}\right) . Then, we study a relaxed version of replicability proposed by Kalavasis et al. [2023] called TV indistinguishability.