Reviews: Learning Temporal Pose Estimation from Sparsely-Labeled Videos

One frame, A contains 2D annotations of human body joints and frame B is unlabeled. The algorithm aims to predict poses for the frame B using only supervision from the frame A. First, a base network (using HRNet backbone [27]) predicts pose heatmaps for both frames. Then, their difference is computed and fed into a stack of residual layers, that predict per-channel offsets which are then used to warp pose heatmaps of the frame B. Finally, they compute the loss between the warped heatmaps and the ground truth heatmaps of the frame A. The warping mechanism is differentiable and is implemented in a similar fashion to the Spatial Transformer Networks by usin Deformable Convolutions [28]. This way the network learns a motion field that warps human body parts from between neighbor frames. The first baseline involves training a standard human pose estimator on the available frames and simply apply the trained detector on the unlabeled frames.