Latent Task-Specific Graph Network Simulators