EgoHumans: An Egocentric 3D Multi-Human Benchmark
Khirodkar, Rawal, Bansal, Aayush, Ma, Lingni, Newcombe, Richard, Vo, Minh, Kitani, Kris
–arXiv.org Artificial Intelligence
We present EgoHumans, a new multi-view multi-human video benchmark to advance the state-of-the-art of egocentric human 3D pose estimation and tracking. Existing egocentric benchmarks either capture single subject or indoor-only scenarios, which limit the generalization of computer vision algorithms for real-world applications. We propose a novel 3D capture setup to construct a comprehensive egocentric multi-human benchmark in the wild with annotations to support diverse tasks such as human detection, tracking, 2D/3D pose estimation, and mesh recovery. We leverage consumer-grade wearable camera-equipped glasses for the egocentric view, which enables us to capture dynamic activities like playing tennis, fencing, volleyball, etc. Furthermore, our multi-view setup generates accurate 3D ground truth even under severe or complete occlusion. The dataset consists of more than 125k egocentric images, spanning diverse scenes with a particular focus on challenging and unchoreographed multi-human activities and fast-moving egocentric views. We rigorously evaluate existing state-of-the-art methods and highlight their limitations in the egocentric scenario, specifically on multi-human tracking. To address such limitations, we propose EgoFormer, a novel approach with a multi-stream transformer architecture and explicit 3D spatial reasoning to estimate and track the human pose. EgoFormer significantly outperforms prior art by 13.6% IDF1 on the EgoHumans dataset.
arXiv.org Artificial Intelligence
Aug-18-2023
- Country:
- Europe
- United Kingdom
- Wales > Ceredigion
- Aberystwyth (0.04)
- England > Cambridgeshire
- Cambridge (0.04)
- Wales > Ceredigion
- Netherlands > North Holland
- Amsterdam (0.04)
- United Kingdom
- Asia
- Myanmar > Tanintharyi Region
- Dawei (0.04)
- Japan > Honshū
- Chūbu > Ishikawa Prefecture > Kanazawa (0.05)
- Myanmar > Tanintharyi Region
- Europe
- Genre:
- Research Report > Promising Solution (0.86)
- Industry:
- Media > Photography (0.46)
- Leisure & Entertainment > Sports (0.34)
- Technology: