The advent of large language models (LLMs) has enabled significant performance gains in the field of natural language processing. However, recent studies have found that LLMs often resort to shortcuts when performing tasks, creating an illusion of enhanced performance while lacking generalizability in their decision rules. This phenomenon introduces challenges in accurately assessing natural language understanding in LLMs. Our paper provides a concise survey of relevant research in this area and puts forth a perspective on the implications of shortcut learning in the evaluation of language models, specifically for NLU tasks. This paper urges more research efforts to be put towards deepening our comprehension of shortcut learning, contributing to the development of more robust language models, and raising the standards of NLU evaluation in real-world scenarios.

Jack Krawczyk discusses how Google Bard helps users connect and communicate -- and what the future holds for the platform. Google AI has officially transitioned into Gemini, an enhanced version of Google's first artificial intelligence system. In a conversation with Fox News Digital, Google AI product lead Jack Krawczyk revealed what's new about Gemini. "It's genuinely an ecosystem that we're going to be building on as a company," he said. Google has announced Gemini Advanced and an app version of its AI tool.

Many people--like, say, journalists--are understandably antsy about what generative artificial intelligence might mean for the future of their profession. Some workers are already living in one potential version of the generative AI future, though: computer programmers. "Developers have arrived in the age of AI," says Thomas Dohmke, CEO of GitHub. "The only question is, how fast do you get on board? Or are you going to be stuck in the past, on the wrong side of the'productivity polarity'?"

When I was a kid, the gadgets in any given movie or TV show were the real stars. John Connor's handheld Atari, Marty McFly's self-lacing Nikes, and about a million different variations on the radio watch were what I dreamed of owning one day. So I feel a certain kinship with YouTuber Abe's Projects, who took his love for fictional gadgets one step further and made a real version of the Pokedex from the Pokemon cartoon. If you weren't the right age in the late 90s, the Pokedex is a handheld computer riffing on the Palm Pilots of the day. Its sole purpose is to identify and catalog Pokemon creatures in the wild.

The news: In the biggest mass-market AI launch yet, Google is rolling out Gemini, its family of large language models, across almost all its products, from Android to the iOS Google app to Gmail to Docs and more. A new subscription plan will also give users access to Gemini Ultra, the most powerful version of the model, for the first time. Why it matters: ChatGPT, released by Microsoft-backed OpenAI just 14 months ago, changed people's expectations of what computers could do. Google has been racing to catch up ever since and unveiled its Gemini family of models in December. By baking Gemini into its ubiquitous tools, it will be hoping to make up any lost ground, and even overtake its rival.

ChatGPT, released by Microsoft-backed OpenAI just 14 months ago, changed people's expectations of what computers could do. Google, which has been racing to catch up ever since, unveiled its Gemini family of models in December. They are multimodal large language models that can interact with you via voice, image, and text. Google claimed that its own benchmarking showed Gemini outperforming GPT-4 on a range of standard tests. But the margins were slim. By baking Gemini into its ubiquitous tools, Google is hoping to make up lost ground and even overtake its rival.

Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. Specifically, an LLM is utilized to carry out a direct mapping from the N-best hypotheses list generated by an ASR system to the predicted output transcription. However, despite its effectiveness, GER introduces extra data uncertainty since the LLM is trained without taking into account acoustic information available in the speech signal. In this work, we aim to overcome such a limitation by infusing acoustic information before generating the predicted transcription through a novel late fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF). UADF is a multimodal fusion approach implemented into an auto-regressive decoding process and works in two stages: (i) It first analyzes and calibrates the token-level LLM decision, and (ii) it then dynamically assimilates the information from the acoustic modality. Experimental evidence collected from various ASR tasks shows that UADF surpasses existing fusion mechanisms in several ways. It yields significant improvements in word error rate (WER) while mitigating data uncertainty issues in LLM and addressing the poor generalization relied with sole modality during fusion. We also demonstrate that UADF seamlessly adapts to audio-visual speech recognition.

Motivated by the success of Transformers when applied to sequences of discrete symbols, token-based world models (TBWMs) were recently proposed as sample-efficient methods. In TBWMs, the world model consumes agent experience as a language-like sequence of tokens, where each observation constitutes a sub-sequence. However, during imagination, the sequential token-by-token generation of next observations results in a severe bottleneck, leading to long training times, poor GPU utilization, and limited representations. To resolve this bottleneck, we devise a novel Parallel Observation Prediction (POP) mechanism. POP augments a Retentive Network (RetNet) with a novel forward mode tailored to our reinforcement learning setting. We incorporate POP in a novel TBWM agent named REM (Retentive Environment Model), showcasing a 15.4x faster imagination compared to prior TBWMs. REM attains superhuman performance on 12 out of 26 games of the Atari 100K benchmark, while training in less than 12 hours. Our code is available at \url{https://github.com/leor-c/REM}.

We study the problem of zero-shot exercise retrieval in the context of online language learning, to give learners the ability to explicitly request personalized exercises via natural language. Using real-world data collected from language learners, we observe that vector similarity approaches poorly capture the relationship between exercise content and the language that learners use to express what they want to learn. This semantic gap between queries and content dramatically reduces the effectiveness of general-purpose retrieval models pretrained on large scale information retrieval datasets like MS MARCO. We leverage the generative capabilities of large language models to bridge the gap by synthesizing hypothetical exercises based on the learner's input, which are then used to search for relevant exercises. Our approach, which we call mHyER, overcomes three challenges: (1) lack of relevance labels for training, (2) unrestricted learner input content, and (3) low semantic similarity between input and retrieval candidates. mHyER outperforms several strong baselines on two novel benchmarks created from crowdsourced data and publicly available data.

Existing Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled directly by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing a variety of daily computer tasks. Finally, we trained a model, ScreenAgent, which achieved computer control capabilities comparable to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code is available at \url{https://github.com/niuzaisheng/ScreenAgent}.