Leveraging real-time eye-tracking, foveated rendering optimizes hardware efficiency and enhances visual quality virtual reality (VR). This approach leverages eye-tracking techniques to determine where the user is looking, allowing the system to render high-resolution graphics only in the foveal region-the small area of the retina where visual acuity is highest, while the peripheral view is rendered at lower resolution. However, modern deep learning-based gaze-tracking solutions often exhibit a long-tail distribution of tracking errors, which can degrade user experience and reduce the benefits of foveated rendering by causing misalignment and decreased visual quality. This paper introduces \textit{FovealNet}, an advanced AI-driven gaze tracking framework designed to optimize system performance by strategically enhancing gaze tracking accuracy. To further reduce the implementation cost of the gaze tracking algorithm, FovealNet employs an event-based cropping method that eliminates over $64.8\%$ of irrelevant pixels from the input image. Additionally, it incorporates a simple yet effective token-pruning strategy that dynamically removes tokens on the fly without compromising tracking accuracy. Finally, to support different runtime rendering configurations, we propose a system performance-aware multi-resolution training strategy, allowing the gaze tracking DNN to adapt and optimize overall system performance more effectively. Evaluation results demonstrate that FovealNet achieves at least $1.42\times$ speed up compared to previous methods and 13\% increase in perceptual quality for foveated output.

Virtual Reality (VR) interfaces are increasingly used as remote visualization media in telerobotics. Remote environments captured through RGB-D cameras and visualized using VR interfaces can enhance operators' situational awareness and sense of presence. However, this approach has strict requirements for the speed, throughput, and quality of the visualized 3D data.Further, telerobotics requires operators to focus on their tasks fully, requiring high perceptual and cognitive skills. This paper shows a work-in-progress framework to address these challenges by taking the human visual system (HVS) as an inspiration. Human eyes use attentional mechanisms to select and draw user engagement to a specific place from the dynamic environment. Inspired by this, the framework implements functionalities to draw users's engagement to a specific place while simultaneously reducing latency and bandwidth requirements.

This article is part of a VB special issue. Read the full series here: The metaverse - How close are we? Some companies believe that the metaverse -- a yet-to-be-realized, internet-like series of connected worlds -- has enormous potential in the enterprise. For example, it could be used to improve work productivity by allowing employees to train or collaborate in workplace-like virtual environments. Or it could host home and office tours, a boon for a real estate market contending with pandemic travel restrictions.

Researchers from the Institute of Electrical and Electronics Engineers (IEEE) have developed a method to increase the authenticity of low-cost, projection-based augmented reality installations, through special glasses that cause projected 3D images to go in and out of focus in the same way that they would if the objects were real, overcoming a critical perceptual hurdle for practical usage of projection systems in controlled environments. The IEEE system recreates depth planes for projected real and CGI imagery that will be superimposed into rooms. In this case, three CGI Stanford bunnies are being superimposed at the same depth plane as three real world objects, and their blurriness is controlled by where the viewer is looking and focusing. The system uses electrically focus-tunable lenses (ETL) embedded into the viewer's glasses (which are in any case necessary to separate the two image streams into a convincing, integrated 3D experience), and which communicate with the projection system, which then automatically changes the level of blurriness of the projected image seen by the viewer. The ETL lenses report back information about the user's focal attention and sets the level of blurriness on a per-plane basis for the rendering of the projected geometry.

If rumors are anything to go by, Sony's next-generation PlayStation VR will feature eye-tracking and will have twice the resolution of the current PSVR. On May 10, Upload VR shared exciting rumors about Sony's plan for its upcoming next-gen VR headset. The report said the VR headset for the PlayStation 5 console will be loaded with features such as inside-out tracking and an increased resolution compared to its predecessor. The report said the rumors were via partners to whom Sony shared details about the product with. The next-gen Sony Play Station VR headset will reportedly have a 4000 x 2000 resolution.

Foveated rendering addresses a growing challenge for VR headsets, rendering sharp details for your eye's visual sweet spot -- the fovea -- and a simpler, blurrier version for your peripheral vision. Now engineers at Facebook Reality Labs have come up with DeepFovea, an AI-assisted alternative that creates "plausible peripheral video" rather than actually rendering accurate peripheral imagery. The new process is known as "foveated reconstruction," and Facebook says it achieves more than 14 times compression on RGB video with no significant degradation in user-perceived quality. When capturing a video stream, DeepFovea samples only 10% of the pixels in each video frame, focusing largely but not exclusively on the area where the user's eye is focused, represented by the lizard head above. By comparison, the peripheral area is sampled only by scattered dots that become less dense further from the eye's focus area.

Note: Moor Insights & Strategy Analyst Anshel Sag made significant contributions to this blog. AR and VR are still in the growth phase of development with billions of R&D and content money being poured in annually. A lot of that money is being invested in technologies to improve the experience as the current experience is, well, rough. There is a multitude of technologies that can help the immersive technology industry to move forward with AR and VR. Some of those technologies have to do with the display or wireless communications to make the experience better than it is today.

When I got one of Tobii's eye-tracking devices a few years ago, I didn't know what to do with it. Tobii bills itself as "the world leader in eye tracking," and as far as I could tell the claim rang true. The device certainly worked, and I could see it being very useful for people who (for any number of reasons) can't use a keyboard and mouse. But Tobii clearly wanted to broaden its ambitions and turn eye-tracking into a more mass-market solution, not an alternate means of interfacing with PCs but a primary means. Ubisoft builds Tobii support into all its games, and I played Assassin's Creed: Rogue with Tobii's eye-tracker swinging the camera wherever I looked.

Ben Dickson is a software engineer and the founder of TechTalks. The concept of measuring and responding to human eye motion, or eye tracking, isn't new, but the past year saw a rising interest in the technology. There have been a slew of acquisitions of eye tracking startups by large firms and the rollout of several devices and software that support eye tracking. "Eye tracking sensors provide two main benefits," says Oscar Werner, vice president of the eye tracking company Tobii Tech. "First, it makes a device aware of what the user is interested in at any given point in time. And second, it provides an additional way to interact with content, without taking anything else away. That means it increases the communication bandwidth between the user and the device."