Learning to Collaborate in Markov Decision Processes