Towards A Unified Agent with Foundation Models