Aligning Agents like Large Language Models