Distributed Occlusion Reasoning for Tracking with Nonparametric Belief Propagation

We describe a three-dimensional geometric hand model suitable for visual trackingapplications. The kinematic constraints implied by the model's joints have a probabilistic structure which is well described by a graphical model. Inference in this model is complicated by the hand's many degrees of freedom, as well as multimodal likelihoods caused by ambiguous image measurements. We use nonparametric belief propagation (NBP)to develop a tracking algorithm which exploits the graph's structure to control complexity, while avoiding costly discretization. While kinematic constraints naturally have a local structure, self-occlusions created by the imaging process lead to complex interpendencies incolor and edge-based likelihood functions. However, we show that local structure may be recovered by introducing binary hidden variables describingthe occlusion state of each pixel. We augment the NBP algorithm to infer these occlusion variables in a distributed fashion, and then analytically marginalize over them to produce hand position estimates whichproperly account for occlusion events. We provide simulations showing that NBP may be used to refine inaccurate model initializations, aswell as track hand motion through extended image sequences.