Appendices A Notes on the MDP formulation