A Framework for Sequential Planning in Multi-Agent Settings