Understanding Cognitive States from Head & Hand Motion Data

The pipeline illustrates the full workflow from data collection in VR, through self-annotation and human baseline evaluation, to modeling and analysis of cognitive states. As virtual reality (VR) and augmented reality (AR) continue to gain popularity, head and hand motion data captured by consumer VR systems have become ubiquitous. Prior work shows such telemetry can be highly identifying and reflect broad user traits, often aligning with intuitive "folk theories" of body language. However, it remains unclear to what extent motion kinematics encode more nuanced cognitive states, such as confusion, hesitation, and readiness, which lack clear correlates with motion. To investigate this, we introduce a novel dataset of head and hand motion with frame-level annotations of these states collected during structured decision-making tasks. Our findings suggest that deep temporal models can infer subtle cognitive states from motion alone, achieving comparable performance with human observers. This work demonstrates that standard VR telemetry contains strong patterns related to users' internal cognitive processes, which opens the door for a new gener- To enhance reproducibility and support future work, we will make our dataset and modeling framework publicly available. Virtual Reality (VR) is rapidly evolving from a specialized tool for simulation and entertainment into a mainstream computing platform for work, education, and social interaction. As users spend more time in these immersive environments, the quality of human-computer interaction becomes paramount. The next generation of VR systems must move beyond explicit, command-based interfaces and develop the capacity for implicit, nuanced understanding. This requires an ability to perceive and adapt to a user's cognitive state in real-time, creating experiences that are more intuitive, supportive, and effective. The key to unlocking this capability lies in decoding the rich, continuous, and often subconscious stream of motion data generated by every user.