Goto

Collaborating Authors

 toca


Accelerating Diffusion Transformers with Token-wise Feature Caching

arXiv.org Artificial Intelligence

Diffusion transformers have shown significant effectiveness in both image and video synthesis at the expense of huge computation costs. To address this problem, feature caching methods have been introduced to accelerate diffusion transformers by caching the features in previous timesteps and reusing them in the following timesteps. However, previous caching methods ignore that different tokens exhibit different sensitivities to feature caching, and feature caching on some tokens may lead to 10$\times$ more destruction to the overall generation quality compared with other tokens. In this paper, we introduce token-wise feature caching, allowing us to adaptively select the most suitable tokens for caching, and further enable us to apply different caching ratios to neural layers in different types and depths. Extensive experiments on PixArt-$\alpha$, OpenSora, and DiT demonstrate our effectiveness in both image and video generation with no requirements for training. For instance, 2.36$\times$ and 1.93$\times$ acceleration are achieved on OpenSora and PixArt-$\alpha$ with almost no drop in generation quality.


Text-only Synthesis for Image Captioning

arXiv.org Artificial Intelligence

From paired image-text training to text-only training for image captioning, the pursuit of relaxing the requirements for high-cost and large-scale annotation of good quality data remains consistent. In this paper, we propose Text-only Synthesis for Image Captioning (ToCa), which further advances this relaxation with fewer human labor and less computing time. Specifically, we deconstruct caption text into structures and lexical words, which serve as the fundamental components of the caption. By combining different structures and lexical words as inputs to the large language model, massive captions that contain various patterns of lexical words are generated. This method not only approaches the target domain but also surpasses it by generating new captions, thereby enhancing the zero-shot generalization ability of the model. Considering the different levels of data access in the real world, we define three synthesis scenarios: cross-domain synthesis, in-domain synthesis, and data-efficient synthesis. Experiments in these scenarios demonstrate the generalizability, transferability and practicability of ToCa with a nearly 5 CIDEr improvement for zero-shot cross-domain captioning and a maximum increase of over 20 CIDEr for data-efficient captioning.


Developing Decision Aids to Enable Human Spaceflight Autonomy

AI Magazine

As NASA explores destinations beyond the Moon, the distance between Earth and spacecraft will increase communication delays between astronauts and Mission Control. Today, astronauts coordinate with Mission Control to request assistance and await approval to perform tasks. Many of these coordination tasks require multiple exchanges of information, (for example, taking turns). In the presence of long communication delays, the length of time between turns may lead to inefficiency, or increased mission risk. Future astronauts will need software-based decision aids to enable them to work autonomously from Mission Control. These tools require the right combination of mission operations functions, for example, automated planning and fault management, troubleshooting recommendations, easy to access information, and just-in-time training. Ensuring these elements are properly designed and integrated requires an integrated human factors approach. This article describes a recent demonstration of autonomous mission operations using a novel software-based decision aid onboard the International Space Station. We describe how this new technology changes the way astronauts coordinate with mission control, and how the lessons learned from these early demonstrations will enable the operational autonomy needed to ensure astronauts can safely journey to Mars, and beyond.