However, crucially, we do not make two assumptions used to derive the AVAE objective. Firstly, we do not assume that the decoder is a one-to-one mapping between latent samples and a corresponding generated sample. Interestingly,theDCIresponsibility matrices dooftenresemble theconditioned response matrices, suggesting that relying on correlations instead of a full causal analysis, can yield similar results. Infact, then the learned causal structure estimated using the latent response matrix may be used in tandem to develop a structure-aware disentanglementmetric. Given the complexity of the underlying data manifold, aviable alternativeisbased on riemanian geometry [78]which has previously been investigated for alternativeprobabilistic models likeGaussian Process regression [79].

Our goal in this work is to generate realistic videos given just one initial frame as input. Existing unsupervised approaches to this task do not consider the fact that a video typically shows a 3D environment, and that this should remain coherent from frame to frame even as the camera and objects move. We address this by developing a model that first estimates the latent 3D structure of the scene, including the segmentation of any moving objects. It then predicts future frames by simulating the object and camera dynamics, and rendering the resulting views. Importantly, it is trained end-to-end using only the unsupervised objective of predicting future frames, without any 3D information nor segmentation annotations. Experiments on two challenging datasets of natural videos show that our model can estimate 3D structure and motion segmentation from a single frame, and hence generate plausible and varied predictions.