Modelling Patient Trajectories Using Multimodal Information