Video Large Language Models (Video-LLMs) excel at understanding videos in-context, provided they have full access to the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free approach compatible with standard Video-LLMs, leveraging three key concepts: 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-based selection allows us to discard up to ~95% of unimportant visual tokens with minimal performance loss; 2) Recurrent processing of past selected tokens to generate temporally coherent understanding of each processed clip; 3) Caption-based question answering for lightweight and accurate responses. Our method achieves state-of-the-art performance on streaming video benchmarks, striking a balance between efficiency and effectiveness.

Barriers to accessing mental health assessments including cost and stigma continues to be an impediment in mental health diagnosis and treatment. Machine learning approaches based on speech samples could help in this direction. In this work, we develop machine learning solutions to diagnose anxiety disorders from audio journals of patients. We work on a novel anxiety dataset (provided through collaboration with Kintsugi Mindful Wellness Inc.) and experiment with several models of varying complexity utilizing audio, text and a combination of multiple modalities. We show that the multi-modal and audio embeddings based approaches achieve good performance in the task achieving an AUC ROC score of 0.68-0.69.

I’ll demonstrate how to extract an audio clip from YouTube, implement speech recognition using OpenAI’s Whisper, and perform speech generation using Tortoise to clone a custom voice.

The Guzheng is a kind of traditional Chinese instruments with diverse playing techniques. Instrument playing techniques (IPT) play an important role in musical performance. However, most of the existing works for IPT detection show low efficiency for variable-length audio and provide no assurance in the generalization as they rely on a single sound bank for training and testing. In this study, we propose an end-to-end Guzheng playing technique detection system using Fully Convolutional Networks that can be applied to variable-length audio. Because each Guzheng playing technique is applied to a note, a dedicated onset detector is trained to divide an audio into several notes and its predictions are fused with frame-wise IPT predictions. During fusion, we add the IPT predictions frame by frame inside each note and get the IPT with the highest probability within each note as the final output of that note. We create a new dataset named GZ_IsoTech from multiple sound banks and real-world recordings for Guzheng performance analysis. Our approach achieves 87.97% in frame-level accuracy and 80.76% in note-level F1-score, outperforming existing works by a large margin, which indicates the effectiveness of our proposed method in IPT detection.

In this project, I use faceswap repo to swap faces between Lukas and Chris who were kind enough to let me play with their pictures for the purpose of this experiment. I'll particularly focus on the effect of data in improving the model. The challenge is clear, we are trying to swap faces between people with different skin tone and facial hair. Usually, you would want to find people with similar features in order to improve the success of your face swapping. In many cases, you swap only the inner part of the face.

A new app from the former head of video-sharing app Vine hopes to repeat the success of the cult social network by making it easier to shoot and edit short clips. Trash hopes that its secret weapon will be "computational cinematography": the app, which entered closed beta on Monday, uses machine learning "to automate the un-fun parts of video editing", automatically processing video to cut together short clips with a consistent mood and feel. A similar approach, computational photography, has already radically changed smartphone photography, enabling features such as the Pixel's Night Sight and iPhone's Portrait Mode. Trash's co-founder, Hannah Donovan, who was Vine's last general manager before the service was shut down by its owner, Twitter, said she hoped the approach would lower the barrier of entry to video editing. "We're analysing the video for a bunch of different things," Donovan said.

We design, conduct and present the results of a highly personalized baseline emotion recognition experiment, which aims to set reliable ground-truth estimates for the subject's emotional state for real-life prediction under similar conditions using a small number of physiological sensors. We also propose an adaptive stimuli-selection mechanism that would use the user's feedback as guide for future stimuli selection in the controlled-setup experiment and generate optimal ground-truth personalized sessions systematically. Initial results are very promising (85% accuracy) and variable importance analysis shows that only a few features, which are easy-to-implement in portable devices, would suffice to predict the subject's emotional state.

There are many speech generation programs that create artificial human speech such as Minitalk and eSpeak, but Google believes they have a system that outperforms current technology by 50%. Deepmind is a Google unit that is working on super-intelligent computers and they've created an artificial intelligence that can closely mimic human speech. The AI, called WaveNet, works by analyzing a human voice's actual sound waves. More archaic speech generators either use short clips of a previously recorded speaker or electronically generate speech based on how certain letter combinations are supposed to be pronounced. The results of both provide highly accurate speech, but they sound robotic and lack the fluidity of human diction.

The AAAI video archive is a central source of information about videotapes and films with information about AI that are stored digitally on other sites or physically in institutional archives. For each video, the archive includes a brief description of the contents and personae, one or more representative short clips for classroom or individual use, and the location of the archival copy (for example, at a university library).