Large language models (LLMs), or foundation models (FMs), are pretrained transformers that coherently complete sentences auto-regressively. In this paper, we show that LLMs can control simplified space systems after some additional training, called fine-tuning. We look at relatively small language models, ranging between 7 and 13 billion parameters. We focus on four problems: a three-dimensional spring toy problem, low-thrust orbit transfer, low-thrust cislunar control, and powered descent guidance. The fine-tuned LLMs are capable of controlling systems by generating sufficiently accurate outputs that are multi-dimensional vectors with up to 10 significant digits. We show that for several problems the amount of data required to perform fine-tuning is smaller than what is generally required of traditional deep neural networks (DNNs), and that fine-tuned LLMs are good at generalizing outside of the training dataset. Further, the same LLM can be fine-tuned with data from different problems, with only minor performance degradation with respect to LLMs trained for a single application. This work is intended as a first step towards the development of a general space systems controller.

Since the emergence of the LLM trend, initiated with the first release of ChatGPT [1], these systems have undergone continuous development and have evolved into multimodal architectures. Multimodal models, such as GPT-4o [2], LLaMA 3.2 [3] and Claude with its latest 3.5 Sonnet model [4], integrate language understanding with non-language capabilities, including vision and audio processing. This progression unlocks new opportunities for developing intelligent agents capable of recognizing and interpreting patterns not only at a semantic level but also through components that can incorporate other types of unstructured data into prompts, significantly expanding their potential applications and impact. Extending these capabilities, Vision-Language Models (VLMs) build on multimodal principles by integrating visual reasoning into the LLM framework. By introducing new tokens into the prompts to process image frames, VLMs enable simultaneous semantic and visual reasoning. This enhancement is particularly valuable in dynamic applications like robotics, where the integration of vision and language reasoning enables systems to generate environment-responsive actions. Such actions, often described as descriptive policies, translate reasoning into meaningful, executable commands. Language models able to generate such commands are usually referred to as "agentic".

This work introduces Transformer-based Successive Convexification (T-SCvx), an extension of Transformer-based Powered Descent Guidance (T-PDG), generalizable for efficient six-degree-of-freedom (DoF) fuel-optimal powered descent trajectory generation. Our approach significantly enhances the sample efficiency and solution quality for nonconvex-powered descent guidance by employing a rotation invariant transformation of the sampled dataset. T-PDG was previously applied to the 3-DoF minimum fuel powered descent guidance problem, improving solution times by up to an order of magnitude compared to lossless convexification (LCvx). By learning to predict the set of tight or active constraints at the optimal control problem's solution, Transformer-based Successive Convexification (T-SCvx) creates the minimal reduced-size problem initialized with only the tight constraints, then uses the solution of this reduced problem to warm-start the direct optimization solver. 6-DoF powered descent guidance is known to be challenging to solve quickly and reliably due to the nonlinear and non-convex nature of the problem, the discretization scheme heavily influencing solution validity, and reference trajectory initialization determining algorithm convergence or divergence. Our contributions in this work address these challenges by extending T-PDG to learn the set of tight constraints for the successive convexification (SCvx) formulation of the 6-DoF powered descent guidance problem. In addition to reducing the problem size, feasible and locally optimal reference trajectories are also learned to facilitate convergence from the initial guess. T-SCvx enables onboard computation of real-time guidance trajectories, demonstrated by a 6-DoF Mars powered landing application problem.

Despite its promise and the tremendous advances in nonlinear optimization solvers in recent years, trajectory optimization has primarily been constrained to offline usage due to the limited compute capabilities of radiation hardened flight computers [3]. However, with a flurry of proposed mission concepts that call for increasingly greater on-board autonomy [4], bridging this gap in the state-of-practice is necessary to allow for scaling current trajectory design techniques for future missions. Recently, researchers have turned to machine learning and data-driven techniques as a promising method for reducing the runtimes necessary for solving challenging constrained optimization problems [5, 6]. Such approaches entail learning what is known as the problem-to-solution mapping between the problem parameters that vary between repeated instances of solving the trajectory optimization problem to the full optimization solution and these works typically use a Deep Neural Network (DNN) to model this mapping [7-9]. Given parameters of new instances of the trajectory optimization problem, this problem-to-solution mapping can be used online to yield candidate trajectories to warm start the nonlinear optimization solver and this warm start can enable significant solution speed ups. One shortcoming of these aforementioned data-driven approaches is that they have limited scope of use and the learned problem-to-solution mapping only applies for one specific trajectory optimization formulation. With a change to the mission design specifications that yields, e.g., a different optimization constraint, a new problem-to-solution mapping has to be learned offline and this necessitates generating a new dataset of solved trajectory optimization problems. To this end, our work explores the use of compositional diffusion modeling to allow for generalizable learning of the problem-to-solution mapping and equip mission designers with the ability to interleave different learned models to satisfy a rich set of trajectory design specifications. Compositional diffusion modeling enables training of a model to both sample and plan from.

Novel view synthesis (NVS) enables to generate new images of a scene or convert a set of 2D images into a comprehensive 3D model. In the context of Space Domain Awareness, since space is becoming increasingly congested, NVS can accurately map space objects and debris, improving the safety and efficiency of space operations. Similarly, in Rendezvous and Proximity Operations missions, 3D models can provide details about a target object's shape, size, and orientation, allowing for better planning and prediction of the target's behavior. In this work, we explore the generalization abilities of these reconstruction techniques, aiming to avoid the necessity of retraining for each new scene, by presenting a novel approach to 3D spacecraft reconstruction from single-view images, DreamSat, by fine-tuning the Zero123 XL, a state-of-the-art single-view reconstruction model, on a high-quality dataset of 190 high-quality spacecraft models and integrating it into the DreamGaussian framework. We demonstrate consistent improvements in reconstruction quality across multiple metrics, including Contrastive Language-Image Pretraining (CLIP) score (+0.33%), Peak Signal-to-Noise Ratio (PSNR) (+2.53%), Structural Similarity Index (SSIM) (+2.38%), and Learned Perceptual Image Patch Similarity (LPIPS) (+0.16%) on a test set of 30 previously unseen spacecraft images. Our method addresses the lack of domain-specific 3D reconstruction tools in the space industry by leveraging state-of-the-art diffusion models and 3D Gaussian splatting techniques. This approach maintains the efficiency of the DreamGaussian framework while enhancing the accuracy and detail of spacecraft reconstructions. The code for this work can be accessed on GitHub (https://github.com/ARCLab-MIT/space-nvs).

This work introduces TrajDiffuser, a compositional diffusion-based flexible and concurrent trajectory generator for 6 degrees of freedom powered descent guidance. TrajDiffuser is a statistical model that learns the multi-modal distributions of a dataset of simulated optimal trajectories, each subject to only one or few constraints that may vary for different trajectories. During inference, the trajectory is generated simultaneously over time, providing stable long-horizon planning, and constraints can be composed together, increasing the model's generalizability and decreasing the training data required. The generated trajectory is then used to initialize an optimizer, increasing its robustness and speed.

Recent trends are emerging in the use of Large Language Models (LLMs) as autonomous agents that take actions based on the content of the user text prompts. We intend to apply these concepts to the field of Guidance, Navigation, and Control in space, enabling LLMs to have a significant role in the decision-making process for autonomous satellite operations. As a first step towards this goal, we have developed a pure LLM-based solution for the Kerbal Space Program Differential Games (KSPDG) challenge, a public software design competition where participants create autonomous agents for maneuvering satellites involved in non-cooperative space operations, running on the KSP game engine. Our approach leverages prompt engineering, few-shot prompting, and fine-tuning techniques to create an effective LLM-based agent that ranked 2nd in the competition.

This escalating trend is projected to continue as multiple companies, including SpaceX, Amazon, and Astra Space, plan to launch large constellations of hundreds to thousands of satellites. The resulting dense and complex operating environment elevates the risk of collisions and debris generation, posing substantial challenges for space operators. Not only does this situation threaten the safety of flight and mission success in the short run, but it also jeopardizes the long-term viability of the LEO environment for scientific, commercial, and national security uses. Hence, understanding and modeling the evolution of the space environment is crucial for ensuring its sustainability and informing strategies for effective space traffic management. A variety of models have emerged to examine this evolution and calculate the orbital capacity, which is referred to as the number of satellites that can feasibly be situated in LEO [1]. Traditional proprietary models, developed by organizations like NASA's LEGEND [2], ESA's DELTA [3], JAXA's IMPACT [4] among others [5, 6], have been complemented by newer open-source initiatives such as the MIT Orbital Capacity Tool (MOCAT) and its various versions [7, 8]. Most of these models operate by propagating all the ASOs forward in time, utilizing physical models of spacecraft dynamics. This methodology incorporates factors such as atmospheric drag, solar radiation pressure, third-body perturbations, and space weather, in addition to simulated collisions and explosions.

Future spacecraft and surface robotic missions require increasingly capable autonomy stacks for exploring challenging and unstructured domains and trajectory optimization will be a cornerstone of such autonomy stacks. However, the nonlinear optimization solvers required remain too slow for use on relatively resource constrained flight-grade computers. In this work, we turn towards amortized optimization, a learning-based technique for accelerating optimization run times, and present TOAST: Trajectory Optimization with Merit Function Warm Starts. Offline, using data collected from a simulation, we train a neural network to learn a mapping to the full primal and dual solutions given the problem parameters. Crucially, we build upon recent results from decision-focused learning and present a set of decision-focused loss functions using the notion of merit functions for optimization problems. We show that training networks with such constraint-informed losses can better encode the structure of the trajectory optimization problem and jointly learn to reconstruct the primal-dual solution while also yielding improved constraint satisfaction. Through numerical experiments on a Lunar rover problem, we demonstrate that TOAST outperforms benchmark approaches in terms of both computation times and network prediction constraint satisfaction.

In this work, we present Transformer-based Powered Descent Guidance (T-PDG), a scalable algorithm for reducing the computational complexity of the direct optimization formulation of the spacecraft powered descent guidance problem. T-PDG uses data from prior runs of trajectory optimization algorithms to train a transformer neural network, which accurately predicts the relationship between problem parameters and the globally optimal solution for the powered descent guidance problem. The solution is encoded as the set of tight constraints corresponding to the constrained minimum-cost trajectory and the optimal final time of landing. By leveraging the attention mechanism of transformer neural networks, large sequences of time series data can be accurately predicted when given only the spacecraft state and landing site parameters. When applied to the real problem of Mars powered descent guidance, T-PDG reduces the time for computing the 3 degree of freedom fuel-optimal trajectory, when compared to lossless convexification, from an order of 1-8 seconds to less than 500 milliseconds. A safe and optimal solution is guaranteed by including a feasibility check in T-PDG before returning the final trajectory.