Encoding Agent Trajectories as Representations with Sequence Transformers