ASDSV: Multimodal Generation Made Efficient with Approximate Speculative Diffusion and Speculative Verification

Diffusion in transformer is central to advances in high-quality multimodal generation but suffer from high inference latency due to their iterative nature. Inspired by speculative decoding's success in accelerating large language models, we propose Approximate Speculative Diffusion with Speculative Verification (ASDSV), a novel method to enhance the efficiency of diffusion models. Adapting speculative execution to diffusion processes presents unique challenges. First, the substantial computational cost of verifying numerous speculative steps for continuous, high-dimensional outputs makes traditional full verification prohibitively expensive. Second, determining the optimal number of speculative steps $K$ involves a trade-off between potential acceleration and verification success rates. To address these, ASDSV introduces two key innovations: 1) A speculative verification technique, which leverages the observed temporal correlation between draft and target model outputs, efficiently validates $K$ speculative steps by only checking the alignment of the initial and final states, significantly reducing verification overhead.