Appendix - Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties

We have released the Physion++ dataset at our project page. We also post a list of all test filenames in the test set (192 entries per physical property). The dataset is organized as follows. The.json file contains the object id and segmentation mask of each instance over time. The.pkl file contains the meta information of the scene, including 1) the dynamic friction, static friction, initial position, initial rotation, mass, elasticity, color, and mesh of each object; 2) the camera matrix and projection matrix for each frame; 3) angular velocities, positions, rotations, and velocities of all objects over time; 4) the collision events, including the object ids, relative velocities, time and states; 5) the trial seed used to generate the video, the label for the video, and'start_frame_for_prediction', which is a timestamp indicating the part of the video before the timestamp is visible, and the part after the timestamp is required to be predicted. This work was done when Mingyu was at MIT. Table 1: Table of open-source code used. We confirm that we bear all responsibility in case of any violation of rights during the collection of the data or other work, and will take appropriate action when needed, e.g. to remove data with such issues. We ensure access to the data and will provide the necessary maintenance.