Realtime Dynamic Gaze Target Tracking and Depth-Level Estimation

Transparent Displays (TDs) are cutting-edge visual technologies that allow users to see digital content superimposed over physical environments with a variety of applications in dynamic Head-Up Displays (HUDs) in vehicles [1, 2, 3], augmented reality glasses [4, 5, 6], and smart windows in commercial buildings [7]. Their ability to blend digital information with the real world offers significant advancements in fields such as navigation, interactive advertising, robotics [8, 9, 10, 11], and immersive user interfaces and feedback [12, 13, 14, 15, 16]. Imagine a transparent display, such as a dynamic HUD in a vehicle, that not only shows essential metrics like speed, fuel levels, and engine status but also overlays navigational cues directly onto the road ahead, highlighting paths, directions, pedestrians, and other vehicles [2, 1, 17]. Beyond practical utilities, such dynamic HUDs could enhance the journey by identifying points of interest, e.g., service stations, or even serve as platforms for entertainment and work-related activities. However, realizing this vision introduces significant challenges, particularly in tracking the user's gaze across an ever-changing array of widgets and information layers projected onto the transparent display. Moreover, the accurate estimation of gaze depth levels is crucial, especially because of the display's transparency and the potential for the human gaze to interact with or pass through specific widgets, necessitating a system that can precisely discern the focus of a user's attention between virtual overlays and real-world objects to enhance both interactivity and safety [18]. The dynamic nature of this problem, coupled with the need for real-time processing, sets a complex problem space for effectively identifying and monitoring what the user is focusing on at any given moment.