ResNetVLLM -- Multi-modal Vision LLM for the Video Understanding Task

--In this paper, we introduce ResNetVLLM (ResNet Vision LLM), a novel cross-modal framework for zero-shot video understanding that integrates a ResNet-based visual encoder with a Large Language Model (LLM. ResNetVLLM addresses the challenges associated with zero-shot video models by avoiding reliance on pre-trained video understanding models and instead employing a non-pretrained ResNet to extract visual features. This design ensures the model learns visual and semantic representations within a unified architecture, enhancing its ability to generate accurate and contextually relevant textual descriptions from video inputs. Our experimental results demonstrate that ResNetVLLM achieves state-of-the-art performance in zero-shot video understanding (ZSVU) on several benchmarks, including MSRVTT -QA, MSVD-QA, TGIF-QA FrameQA, and ActivityNet-QA. Large language models (LLMs) [1]-[6] have advanced natural language understanding tasks, demonstrating exceptional abilities in comprehending human intentions and interactions. Building on the progress of LLMs, multi-modal LLMs (MLLMs) [7]-[10] have furthered vision-language learning by integrating visual encoders with LLMs and fine-tuning them on language-image instruction-following data. Recently, there has been a surge in video understanding models that leverage LLMs.