Recent contrastive language image pre-training has led to learning highly transferable and robust image representations. However, adapting these models to video domain with minimal supervision remains an open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt an image CLIP model to the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating in an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. A large language model aware of actions and their attributes generates the relevant textual prompts. We introduce two train objectives, concept distillation and concept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.

Appendix382 AAdditional Experiments3383 A.1 Experiments on the ETT datasets384 In the main body, we present a comparison of the benchmark methods on the ETTm2 dataset. In this385 section, we extend our analysis to the remaining three ETT datasets, namely ETTh1, ETTh2, and386 ETTm1, as summarized in Table 7. Our experimental results reveal that Basisformer outperforms all387 other methods in terms of MSE and MAE. In all experiments, lower MSE values indicate better model performance, and we present the best results in boldface. Experimental results with longer length input setting391 Throughout our research, we maintain consistency in our experimental settings by fixing the input392 length to be 96(with a reduced input length of 36for the illness dataset), instead of using a longer393 length.

We follow the notation of [20, 22] with the exception that we use X for the data matrix rather than A, with new data X+ of dimension n b rather than m l. Here, we focus on the updates of the left singular subspace spanned. Details for the right singular subspace are similar and covered in the original works. Matrix sizes are listed for convenience in Table 2. Table 2: Matrix dimensions for incremental SVD. The goal of the algorithm is to maintain an approximation of the data up to the present moment as X QRW> (8) with Q and W orthogonal but R not necessarily diagonal.

Neural Information Processing Systems http://nips.cc/