Modern wearables like the Apple Watch use sensors like gyros and accelerometers to detect hand movements. Those components allow them to turn on their displays when you lift your wrist, as well as to ensure you've properly washed your hand. But thanks to the work of a joint team of researchers at Cornell Unversity and the University of Wisconsin-Madison, future wearables could offer more nuanced hand detection. The team has developed a device they call the FingerTrak. The wearable employs four low-resolution thermal sensors, each the size of about a pea, to monitor the contours of the wrist.

As virtual and augmented reality steadily advance in both visual fidelity and headset comfort, researchers continue to work on input solutions that will feel more natural than holding controllers. Today, a group of researchers announced FingerTrak, a wristband-based solution that uses thermal cameras to track hand movements in 3D, abstracting 20 finger joint positions from contours on the wearer's wrist. Developed by Cornell University's SciFi Lab with assistance from University of Wisconsin, Madison researchers, FingerTrak uses a deep neural network to stitch together input from three or four miniature thermal cameras mounted around the wrist, collectively capturing an entire hand pose. Using silhouettes generated by the cameras, backbone and regression networks estimate fingertip and joint positions, and though the results aren't perfect, they could be used for some forms of VR and AR input. Other potential applications for FingerTrak include human-robot interaction and control, sign language translation, and mobile health, including early detection of degenerative diseases such as Parkinson's and Alzheimer's.