Reinforcement Learning in Markovian and Non-Markovian Environments