Recent efforts have shown promising results in this direction by making use of depth-wiserecurrentnetworks.

To enable communication between points at lower layers, we also add pooling and expansion layers between the ResNet-blocks. The encoder is a bidirectional LSTM [1]. The decoder is asimilar LSTM that generates avector from the previous token ofthe output sequence. In general, the whole training process is split into 3 stages.

We bear all responsibility in case of violation of rights. The rest of this supplementary document is organized as the following. Next, in Section C, we describe the user interface for dataset collection. On average, we can obtain 29.4 descriptions per video, highlighting the advantage of our First, CLEVRER-Humans contains dense annotations of causal relations between physical events. The outer circle represents the general event families. We have lemmatized all verbs to remove the tense.

Building machines that can reason about physical events and their causal relationships is crucial for flexible interaction with the physical world.