Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHighlights) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, Moment-DETR substantially outperforms previous methods.

Policies generated by Reinforcement Learning (RL) algorithms can be difficult to describe to users, as they result from the interplay between complex reward structures and neural network-based representations. This combination often leads to unpredictable behaviors, making policies challenging to analyze and posing significant obstacles to fostering human trust in real-world applications. Global policy summarization methods aim to describe agent behavior through a demonstration of actions in a subset of world-states. However, users can only watch a limited number of demonstrations, restricting their understanding of policies. Moreover, those methods overly rely on user interpretation, as they do not synthesize observations into coherent patterns. In this work, we present SySLLM (Synthesized Summary using LLMs), a novel method that employs synthesis summarization, utilizing large language models' (LLMs) extensive world knowledge and ability to capture patterns, to generate textual summaries of policies. Specifically, an expert evaluation demonstrates that the proposed approach generates summaries that capture the main insights generated by experts while not resulting in significant hallucinations. Additionally, a user study shows that SySLLM summaries are preferred over demonstration-based policy summaries and match or surpass their performance in objective agent identification tasks.

We kicked off 2022 with a wonderful post from a fifth grader, Isabella, who (via her father, Rod Fuentes) wrote up a report of her science fair project on optimizing garbage routes in order to reduce litter in her city using computer vision. The final product was a heat map to be offered up to city council for implementation. I particularly loved the way Isabella combined her love for our planet and coding to improve her space, and I especially enjoy reading posts from our young authors. I always appreciate posts that touch on the more ethical aspects of data science, and Aisulu Omar did a wonderful job of shedding light on the importance of representation within data itself, the workplace, and how the two are connected. Addressing systemic biases within data science is so important and this thorough write-up was very effective in breaking down what one needs to know.

Recent metrics for natural language generation rely on pre-trained language models, for instance BERTScore, BLEURT, and COMET. These metrics achieve a high correlation with human evaluations on standard benchmarks. However, it is unclear how these metrics perform for styles and domains that aren't well represented in their training data. In other words, are these metrics robust? The authors found that BERTScore isn't robust to character-level perturbations.

Originally published on Towards AI the World's Leading AI and Technology News and Media Company. If you are building an AI-related product or service, we invite you to consider becoming an AI sponsor. At Towards AI, we help scale AI and technology startups. Let us help you unleash your technology to the masses. I recently applied for jobs in the Data Science space, and while the titles and descriptions were different, the skillsets and responsibilities were the same.

This story will explore how we can reason from and model graphs using labels via Supervised and Semi-Supervised Learning. I'm going to be using a MET Art Collections dataset that will build on my previous parts on Metrics, Unsupervised Learning, and more. Be sure to check out the previous story before this one to keep up on some of the pieces as I won't cover all concepts again in this one: The easiest approach to conduct Supervised Learning is to use graph measures as features in a new dataset or in addition to an existing dataset. I have seen this method yield positive results for modeling tasks, but it can be really dependent on 1. how you model as a graph (what are the inputs, outputs, edges, etc.) and 2. which metrics to use. Depending on the prediction task, we could compute node-level, edge-level, and graph-level metrics.

This weekend the Riyadh Global Summit For Artificial Intelligence In Healthcare took place in The Kingdom of Saudi Arabia. Leading AI scientists, Pharma executives, and business leaders from around the world came together to explore the potential of AI, machine learning, and deep learning in healthcare. Distinguished speakers attended from The Kingdom of Saudi Arabia, USA, UAE, UK, Canada, Hong Kong, Japan, France, Denmark, and India. Speakers described how AI-powered solutions can improve diagnosis, monitoring, clinical workflow augmentation, and hospital optimization and provided insights into the various AI techniques that can make a significant impact on traditional healthcare structures including natural language processing, data analytics, and machine learning. The Summit included roundtable discussions on the future potential of AI, quantum computing, and blockchain technologies in healthcare.

Highlight: In this post, we will be discussing Variational Autoencoders (VAE). In order to fully understand the underlying ideas, we... Highlight: Over the past few years in machine learning we've seen dramatic progress in the field of generative models. Highlights: GANs and classical Deep Learning methods (classification, object detection) are similar, but they are also fundamentally different in nature.... How did famous tennis players respond to the Djokovic visa saga? Sportsmanship in Tennis as revealed by Artificial Intelligence Software.What famous tennis players REALLY think and FEEL? Highlights: Is your goal to do face recognition in photographs or in videos?

Google Assistant wants to tell you some good news. A new skill aims to give users a reprieve from the oft-depressing daily news cycle by making it easier for them to find more uplifting headlines. Now, users can ask Google Assistant to'Tell me something good,' and it will trigger a'daily dose of good news,' according to the search giant. Google says the skill is launching as an'experimental feature' that's now available on any devices that are equipped with Assistant, such as phones, smart displays and the Google Home, the firm's voice-activated smart speaker. Assistant will serve up stories that are primarily focused around people who are doing things to help their communities and the world, Google explained.

In the spirit of Christmas, we're going to count down to the new year with the 12 Days of AI, bringing you a new, festive AI post every day! What better way to kick off 2018 than to look back at the RE•WORK highlights of 2017 and celebrate some of our successes of the past 12 months. This year saw RE•WORK hosting more events and bringing our globally renowned Summits to new locations. Our first ever Canadian Summit this year took place in Montreal, the'Silicon Valley of AI', and was one of our biggest events to date with over 600 attendees over the two days. We were fortunate enough to be joined by the'Godfathers of AI', Yoshua Bengio, Yann LeCun and Geoffrey Hinton who appeared on a panel together for the first time ever.