DAM: Dynamic Adapter Merging for Continual Video QA Learning

We present a parameter-efficient method for continual video question-answering (VidQA) learning. M, uses the proposed Dynamic Adapter Merging to (i) mitigate catastrophic forgetting, (ii) enable efficient adaptation to continually arriving datasets, (iii) handle inputs from unknown datasets during inference, and (iv) enable knowledge sharing across similar dataset domains. Given a set of continually streaming VidQA datasets, we sequentially train dataset-specific adapters for each dataset while freezing the parameters of a large pretrained video-language backbone. During inference, given a videoquestion sample from an unknown domain, our method first uses the proposed non-parametric router function to compute a probability for each adapter, reflecting how relevant that adapter is to the current video-question input instance. Subsequently, the proposed dynamic adapter merging scheme aggregates all the adapter weights into a new adapter instance tailored for that particular test sample to compute the final VidQA prediction, mitigating the impact of inaccurate router predictions and facilitating knowledge sharing across domains. M model outperforms prior state-of-the-art continual learning approaches by 9.1% while exhibiting 1.9% less forgetting on 6 VidQA datasets spanning various domains. M to continual image classification and image QA and outperform prior methods by a large margin. The code is publicly available at: https://github.com/klauscc/DAM. The role of video in our lives has increased tremendously over the recent years, with millions of hours of video uploaded to the Internet daily. Due to such rapid video growth and the emergence of video-language models (Yu et al., 2021; Yang et al., 2022; Cheng et al., 2023; Wang et al., 2023d; Pramanick et al., 2023b;a), video question-answering (VidQA) has become one of the most important tasks in video understanding. However, modern VidQA models often assume static conditions with fixed training datasets. In contrast, many real-world applications increasingly demand adaptability to distribution shifts of continually arriving datasets. For instance, a VidQA model trained only on movie videos may struggle when questioned about instructional or social media videos due to stark domain disparities.