Goto

Collaborating Authors

 human pilot


Next-Generation LLM for UAV: From Natural Language to Autonomous Flight

Yuan, Liangqi, Deng, Chuhao, Han, Dong-Jun, Hwang, Inseok, Brunswicker, Sabine, Brinton, Christopher G.

arXiv.org Artificial Intelligence

Abstract--With the rapid advancement of Large Language Models (LLMs), their capabilities in various automation domains, particularly Unmanned Aerial V ehicle (UA V) operations, have garnered increasing attention. Current research remains predominantly constrained to small-scale UA V applications, with most studies focusing on isolated components such as path planning for toy drones, while lacking comprehensive investigation of medium-and long-range UA V systems in real-world operational contexts. Larger UA V platforms introduce distinct challenges, including stringent requirements for airport-based take-off and landing procedures, adherence to complex regulatory frameworks, and specialized operational capabilities with elevated mission expectations. LV system processes natural language instructions to orchestrate short-, medium-, and long-range UA V missions through five key technical components: (i) LLM-as-Parser for instruction interpretation, (ii) Route Planner for Points of Interest (POI) determination, (iii) Path Planner for waypoint generation, (iv) Control Platform for executable trajectory implementation, and (v) UA V monitoring. We demonstrate the system's feasibility through three representative use cases spanning different operational scales: multi-UA V patrol, multi-POI delivery, and multi-hop relocation. Beyond the current implementation, we establish a five-level automation taxonomy that charts the evolution from current LLM-as-Parser capabilities (Level 1) to fully autonomous LLMas-Autopilot systems (Level 5), identifying technical prerequisites and research challenges at each stage. The rise of Large Language Models (LLMs) has transformed numerous domains, such as mobile services, vehicles, and robotics [1]-[3]. These fields have become increasingly intelligent and user-friendly through LLM integration, enabling command and control through natural language. Equal contribution L. Y uan and C. G. Brinton are with the School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA. C. Deng and I. Hwang are with the School of Aeronautics and Astronautics, Purdue University, West Lafayette, IN 47907, USA. Han is with the Department of Computer Science and Engineering, Y onsei University, Seoul, South Korea. E-mail: djh@yonsei.ac.kr S. Brunswicker is with the Polytechnic Institute, Purdue University, West Lafayette, IN 47907, USA. LLMs fulfill diverse roles within these systems. LLM-as-Router can orchestrate task allocation and model selection for human pilots, LLM-as-Agent can execute actions on behalf of humans, and LLM-as-Judge can conduct evaluations in place of human judgment.


Reinforcement Learning-Based Monocular Vision Approach for Autonomous UAV Landing

Houichime, Tarik, Amrani, Younes EL

arXiv.org Artificial Intelligence

This paper introduces an innovative approach for the autonomous landing of Unmanned Aerial Vehicles (UAVs) using only a front-facing monocular camera, therefore obviating the requirement for depth estimation cameras. Drawing on the inherent human estimating process, the proposed method reframes the landing task as an optimization problem. The UAV employs variations in the visual characteristics of a specially designed lenticular circle on the landing pad, where the perceived color and form provide critical information for estimating both altitude and depth. Reinforcement learning algorithms are utilized to approximate the functions governing these estimations, enabling the UAV to ascertain ideal landing settings via training. This method's efficacy is assessed by simulations and experiments, showcasing its potential for robust and accurate autonomous landing without dependence on complex sensor setups. This research contributes to the advancement of cost-effective and efficient UAV landing solutions, paving the way for wider applicability across various fields.


UAV-VLRR: Vision-Language Informed NMPC for Rapid Response in UAV Search and Rescue

Yaqoot, Yasheerah, Mustafa, Muhammad Ahsan, Sautenkov, Oleg, Tsetserukou, Dzmitry

arXiv.org Artificial Intelligence

Abstract--Emergency search and rescue (SAR) operations often require rapid and precise target identification in complex environments where traditional manual drone control is inefficient. This system consists of two aspects: 1) A multimodal system which harnesses the power of Visual Language Model (VLM) and the natural language processing capabilities of ChatGPT-4o (LLM) for scene interpretation. This work aims at improving response times in emergency SAR operations by providing a more intuitive and natural approach to the operator to plan the SAR mission while allowing the drone to carry out that mission in a rapid and safe manner. When tested, our approach was faster on an average by 33.75% when compared with an off-the-shelf autopilot and 54.6% when compared with a human pilot. Search and rescue (SAR) operations in disaster-stricken and hazardous environments require fast and efficient situational assessment to locate survivors and critical infrastructure.


Is this helicopter that can fly itself the answer to ending chopper crashes?

FOX News

Kurt "CyberGuy" Knutsson discusses a craft that can fly autonomously without any human intervention. Imagine a helicopter that can take off, fly and land without a human pilot. CLICK TO GET KURT'S FREE CYBERGUY NEWSLETTER WITH SECURITY ALERTS, QUICK VIDEO TIPS, TECH REVIEWS, AND EASY HOW-TO'S TO MAKE YOU SMARTER The R550X is a revolutionary helicopter from Rotor Technologies. It is special because it is the first of its kind to be designed for civilian use, not military or law enforcement. It can perform a variety of missions, such as crop spraying, cargo delivery, firefighting, surveillance, inspection, mapping, surveying, research, exploration, entertainment, and more.


Towards Cooperative Flight Control Using Visual-Attention

Yin, Lianhao, Chahine, Makram, Wang, Tsun-Hsuan, Seyde, Tim, Liu, Chao, Lechner, Mathias, Hasani, Ramin, Rus, Daniela

arXiv.org Artificial Intelligence

The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.


AI-powered drone beats human champion pilots

The Guardian

Having trounced humans at everything from chess and Go, to StarCraft and Gran Turismo, artificial intelligence (AI) has raised its game and laid waste world champions at a physical sport. The latest mortals to feel the sting of AI-induced defeat are three expert drone racers who were beaten by an algorithm that learned to fly a drone around a 3D race course at breakneck speeds without crashing. Developed by researchers at the University of Zurich, the Swift AI won 15 out of 25 races against world champions and clocked the fastest lap on a course where drones reach speeds of 50mph (80km/h) and endure accelerations up to 5g, enough to make many people black out. "Our result marks the first time that a robot powered by AI has beaten a human champion in a real physical sport designed for and by humans," said Elia Kaufmann, a researcher who helped to develop Swift. First-person view drone racing involves flying a drone around a course dotted with gates that must be passed through cleanly to avoid a crash.


Whole-Body Dynamic Telelocomotion: A Step-to-Step Dynamics Approach to Human Walking Reference Generation

Colin, Guillermo, Byrnes, Joseph, Sim, Youngwoo, Wensing, Patrick, Ramos, Joao

arXiv.org Artificial Intelligence

Teleoperated humanoid robots hold significant potential as physical avatars for humans in hazardous and inaccessible environments, with the goal of channeling human intelligence and sensorimotor skills through these robotic counterparts. Precise coordination between humans and robots is crucial for accomplishing whole-body behaviors involving locomotion and manipulation. To progress successfully, dynamic synchronization between humans and humanoid robots must be achieved. This work enhances advancements in whole-body dynamic telelocomotion, addressing challenges in robustness. By embedding the hybrid and underactuated nature of bipedal walking into a virtual human walking interface, we achieve dynamically consistent walking gait generation. Additionally, we integrate a reactive robot controller into a whole-body dynamic telelocomotion framework. Thus, allowing the realization of telelocomotion behaviors on the full-body dynamics of a bipedal robot. Real-time telelocomotion simulation experiments validate the effectiveness of our methods, demonstrating that a trained human pilot can dynamically synchronize with a simulated bipedal robot, achieving sustained locomotion, controlling walking speeds within the range of 0.0 m/s to 0.3 m/s, and enabling backward walking for distances of up to 2.0 m. This research contributes to advancing teleoperated humanoid robots and paves the way for future developments in synchronized locomotion between humans and bipedal robots.


The next world power will be the first to harness the power of AI, former defense official argues in new book

#artificialintelligence

The global battle for AI dominance is underway, according to author Paul Scharre, a former Army Ranger and current VP and director of studies at the Center for New American Security -- a think tank specializing in national security issues. Scharre previously served as a strategic planner at the Office of the Secretary of Defense, working to establish policies on unmanned and autonomous systems and emerging weapons technologies, and established DOD policies on intelligence, surveillance, and reconnaissance programs. In his latest book, "Four Battlegrounds: Power in the Age of Artificial Intelligence," Scharre explores how the international battle for the most powerful AI technology is changing global power dynamics. That battle, he says, is a global competition to seek the best and most efficient data, computing hardware, human talent, and institutions adopting AI technology -- which will determine the next global superpower. In your new book, you argue there's a battle for global power going on in the form of a revolution brought about by artificial intelligence.


US military jet flown by AI for 17 hours: Should you be worried?

FOX News

Jets can be flown by A.I. and can even take off, land and participate in dogfights. Yes, you read the headline correctly. The United States Defense Department recently confirmed that artificial intelligence successfully flew a jet similar to an F-16 for 17 hours straight. The jet was flown over a series of 12 flights back in December 2022 at the Edwards Air Force Base in Kern County, California. CLICK TO GET KURT'S CYBERGUY NEWSLETTER WITH QUICK TIPS, TECH REVIEWS, SECURITY ALERTS AND EASY HOW-TO'S TO MAKE YOU SMARTER The Defense Department used an experimental plane called the Vista X-62A for the flights.


University of Zurich Develops AI Racing Drone, Pits it Against Human Pilots - TechEBlog

#artificialintelligence

There's ion propulsion drones, and then this AI racing drone, developed by University of Zurich researchers. Human drone pilots were invited to the Robotics and Perception Group for a friendly race, with each one getting pit against various AI drones, starting with one using 36 tracking cameras. The camera is used to capture 400 fps of video, in which the AI drone uses in combination with four tracking markers. This footage is then sent to a vision and navigation system capable of translating it into flight commands. These are then sent to the drone in real-time over a wireless connection.