We present a VR-based teleoperation system for multirotor flight that renders a third-person view (TPV) of the vehicle together with a live 3D reconstruction of its surroundings. The system runs on an embedded GPU (Jetson Orin NX) with ROS2-WebXR integration and streams geometry and video to a headset for closed-loop control in previously unmapped spaces. We implement a first-person video (FPV) baseline and perform matched trials with two pilots in unmapped indoor spaces. Quantitative metrics are reported from repeated trials with one pilot (N=8). TPV achieved task time comparable to FPV while improving proximal obstacle awareness (minimum obstacle distance +0.20m) and reducing contacts. These results indicate that TPV can preserve control quality while exposing hazards less visible in FPV, supporting safer teleoperation in unknown environments.

-- Mixed Reality (MR) interfaces have been extensively explored for controlling mobile robots, but there is limited research on their application to managing teams of robots. This paper presents HORUS: Holistic Operational Reality for Unified Systems, a Mixed Reality interface offering a comprehensive set of tools for managing multiple mobile robots simultaneously. HORUS enables operators to monitor individual robot statuses, visualize sensor data projected in real time, and assign tasks to single robots, subsets of the team, or the entire group, all from a Mini-Map (Ground Station). The interface also provides different teleoperation modes: a mini-map mode that allows teleoperation while observing the robot model and its transform on the mini-map, and a semi-immersive mode that offers a flat, screen-like view in either single or stereo view (3D). We conducted a user study in which participants used HORUS to manage a team of mobile robots tasked with finding clues in an environment, simulating search and rescue tasks. This study compared HORUS's full-team management capabilities with individual robot teleoperation. The experiments validated the versatility and effectiveness of HORUS in multi-robot coordination, demonstrating its potential to advance human-robot collaboration in dynamic, team-based environments.

This paper introduces IRIS, an immersive Robot Interaction System leveraging Extended Reality (XR), designed for robot data collection and interaction across multiple simulators, benchmarks, and real-world scenarios. While existing XR-based data collection systems provide efficient and intuitive solutions for large-scale data collection, they are often challenging to reproduce and reuse. This limitation arises because current systems are highly tailored to simulator-specific use cases and environments. IRIS is a novel, easily extendable framework that already supports multiple simulators, benchmarks, and even headsets. Furthermore, IRIS is able to include additional information from real-world sensors, such as point clouds captured through depth cameras. A unified scene specification is generated directly from simulators or real-world sensors and transmitted to XR headsets, creating identical scenes in XR. This specification allows IRIS to support any of the objects, assets, and robots provided by the simulators. In addition, IRIS introduces shared spatial anchors and a robust communication protocol that links simulations between multiple XR headsets. This feature enables multiple XR headsets to share a synchronized scene, facilitating collaborative and multi-user data collection. IRIS can be deployed on any device that supports the Unity Framework, encompassing the vast majority of commercially available headsets. In this work, IRIS was deployed and tested on the Meta Quest 3 and the HoloLens 2. IRIS showcased its versatility across a wide range of real-world and simulated scenarios, using current popular robot simulators such as MuJoCo, IsaacSim, CoppeliaSim, and Genesis. In addition, a user study evaluates IRIS on a data collection task for the LIBERO benchmark. The study shows that IRIS significantly outperforms the baseline in both objective and subjective metrics.

Abstract--The deployment of large language models (LLMs) on extended reality (XR) devices has great potential to advance the field of human-AI interaction. In case of direct, on-device model inference, selecting the appropriate model and device for specific tasks remains challenging. In this paper, we deploy 17 LLMs across four XR devices--Magic Leap 2, Meta Quest 3, Vivo X100s Pro, and Apple Vision Pro--and conduct a comprehensive evaluation. We devise an experimental setup and evaluate performance on four key metrics: performance consistency, processing speed, memory usage, and battery consumption. For each of the 68 model-device pairs, we assess performance under varying string lengths, batch sizes, and thread counts, analyzing the tradeoffs for real-time XR applications. We finally propose a unified evaluation method based on the Pareto Optimality theory to select the optimal device-model pairs from the quality and speed objectives. We believe our findings offer valuable insight to guide future optimization efforts for LLM deployment on XR devices. Our evaluation method can be followed as standard groundwork for further research and development in this emerging field. All supplemental materials are available at nanovis.org/Loxr.html. These models are capable of describing a wide variety of topics, respond at various levels of abstraction, and communicate effectively in multiple languages. They have proven capable of providing users with accurate and contextually appropriate responses. LLMs have quickly found applications in tasks such as spelling and grammar correction [2], generating text on specified topics [3], integration into automated chatbot services, and even generating source code from loosely defined software specifications [4]. Research on language models, and on their multimodal variants integrating language and vision or other technologies has recently experienced rapid growth. For instance, in computer vision, language models are combined with visual signals to achieve tasks such as verbal scene description and even open-world scenegraph generation [5]. These technologies enable detailed interpretation of everyday objects, inference of relationships among them, and estimates of physical properties like size, weight, distance, and speed. In user interaction and visualization research, LLMs serve as verbal interfaces to control software functionality or adjust visualization parameters [6], [7]. Through prompt engineering or fine-tuning, loosely defined text can be translated into specific commands that execute desired actions within a system, supported by language model APIs. The capabilities of language models continue to improve significantly from one version to the next. Xinyu Liu is with King Abdullah University of Science and T echnology (KAUST), Saudi Arabia, and also with University of Electronic Science and T echnology of China, Chengdu, China.

Abstract: Extended Reality (XR) technologies are gaining traction as effective tools for medical training and procedural guidance, particularly in complex cardiac interventions. This paper presents a novel system for real-time 3D tracking and visualization of intracardiac echocardiography (ICE) catheters, with precise measurement of the roll angle. The system's data is integrated into an interactive Unity-based environment, rendered through the Meta Quest 3 XR headset, combining a dynamically tracked catheter with a patient-specific 3D heart model. This immersive environment allows the testing of the importance of 3D depth perception, in comparison to 2D projections, as a form of visualization in XR. Our experimental study, conducted using the ICE catheter with six participants, suggests that 3D visualization is not necessarily beneficial over 2D views offered by the XR system; although all cardiologists saw its utility for pre-operative training, planning, and intra-operative guidance. The proposed system qualitatively shows great promise in transforming catheter-based interventions, particularly ICE procedures, by improving visualization, interactivity, and skill development. Keywords: Percutaneous Cardiac Intervention, Extended Reality, Computer Vision, 3D visualization, ICE catheter, Roll Angle 1. INTRODUCTION Minimally invasive interventions (MII) have revolutionized the field of cardiac care, offering patients reduced recovery times, lower risks of complications, and shorter hospital stays compared to traditional open-heart surgeries. These procedures, such as percutaneous cardiac interventions, rely on the precise navigation of catheters through complex vascular structures and heart chambers[1-6].

Kurt Knutsson looks ahead to the seven emerging trends and innovations in tech that will no doubt transform our lives over the next year. Have you ever wondered what the future will look like? Well, you don't have to wait too long, because 2024 is going to be a year full of amazing innovations that will blow your mind. Here are seven emerging trends and innovations in tech that will no doubt transform our lives over the next year. CLICK TO GET KURT'S FREE CYBERGUY NEWSLETTER WITH SECURITY ALERTS, QUICK VIDEO TIPS, TECH REVIEWS, AND EASY HOW-TO'S TO MAKE YOU SMARTER AI is everywhere, from our daily gadgets like smartphones and smart speakers, to our smart homes that can adjust the temperature, lighting, and security according to our preferences.

The Meta Quest 3 is here, and it's the best standalone VR headset we've ever seen. But is that enough to make people care about virtual reality? In this episode, Devindra and Senior Writer Sam Rutherford chat with Tested's Norm Chan about the Quest 3 and Meta's mixed reality future. While the company's vision of the metaverse is pretty sterile, it's still nice to see Meta learning from the mistakes of the Quest Pro. Sam also dives into his Pixel 8 and Pixel 8 Pro reviews, as well as his thoughts about the Pixel Watch 2. We also dive into Wired's retraction of an op-ed claiming that Google manipulated your search queries, as well as Twitter/X's complete inability to deliver accurate news during the Hamas and Israel conflict.

Undeterred by its many detractors, Meta is still trying to make the metaverse happen. This week, the company held its annual Connect developer conference at its headquarters in Menlo Park, California. Meta CEO Mark Zuckerberg took to the stage to announce a new mixed reality headset, the Meta Quest 3, as well as new smart glasses made by Ray-Ban that let the wearer livestream videos and interact with an AI-powered voice chatbot. Meta also showed off an array of celebrity-infused AI chatbots that can mimic big-name folks like Snoop Dogg and Kendall Jenner. You'd be forgiven for thinking all this feels a little bit like an episode of Black Mirror.

Meta has just wrapped up its 2023 Connect keynote. As promised, the company had a lot more to share about its Meta Quest 3 headset. It also announced the latest pair of smart glasses it created in collaboration with Ray-Ban. In an astoundingly shocking turn of events, Meta CEO Mark Zuckerberg also had some AI updates to discuss. Meta first showed off the Quest 3 back in June to preempt Apple's announcement of the Vision Pro. However, we had to wait a few months to get all of the details about Meta's mixed reality headset (which we've already had some hands-on time with).

Last year Meta caught a bit of backlash when it released the Quest Pro. It was too expensive, it had a number of features people didn't really want, and there just weren't enough apps that fully utilized its hardware. But today at Connect, Meta is announcing the follow-up to the most popular VR headset on the market and I think the Quest 3 is exactly what people have been waiting for. The headset's facelift includes a new y-shaped headband that offers better support along with a fresh row of sensors in front. There are now two full-color cameras that provide sharper pass-through vision along with a depth sensor in the middle that can automatically map your room and detect nearby objects like tables and chairs.