Language model (LM) re-rankers are used to refine retrieval results for retrieval-augmented generation (RAG). They are more expensive than lexical matching methods like BM25 but assumed to better process semantic information. To understand whether LM re-rankers always live up to this assumption, we evaluate 6 different LM re-rankers on the NQ, LitQA2 and DRUID datasets. Our results show that LM re-rankers struggle to outperform a simple BM25 re-ranker on DRUID. Leveraging a novel separation metric based on BM25 scores, we explain and identify re-ranker errors stemming from lexical dissimilarities. We also investigate different methods to improve LM re-ranker performance and find these methods mainly useful for NQ. Taken together, our work identifies and explains weaknesses of LM re-rankers and points to the need for more adversarial and realistic datasets for their evaluation.

Retrieval-augmented generation (RAG) helps address the limitations of the parametric knowledge embedded within a language model (LM). However, investigations of how LMs utilise retrieved information of varying complexity in real-world scenarios have been limited to synthetic contexts. We introduce DRUID (Dataset of Retrieved Unreliable, Insufficient and Difficult-to-understand contexts) with real-world queries and contexts manually annotated for stance. The dataset is based on the prototypical task of automated claim verification, for which automated retrieval of real-world evidence is crucial. We compare DRUID to synthetic datasets (CounterFact, ConflictQA) and find that artificial datasets often fail to represent the complex and diverse real-world context settings. We show that synthetic datasets exaggerate context characteristics rare in real retrieved data, which leads to inflated context utilisation results, as measured by our novel ACU score. Moreover, while previous work has mainly focused on singleton context characteristics to explain context utilisation, correlations between singleton context properties and ACU on DRUID are surprisingly small compared to other properties related to context source. Overall, our work underscores the need for real-world aligned context utilisation studies to represent and improve performance in real-world RAG settings.

Demand for AI-driven virtual assistants and conversational automation is on the rise as focus shifts toward business outcomes and digital engagement. Join us and our partners at Office Samurai for this webinar to see how we're pushing automation to the next level.

Assassin's Creed Valhalla's first major expansion, Wrath of the Druids, will come out a couple of weeks later than expected. It was supposed to be available on April 29th, but now Ubisoft has announced on Twitter that it will be released on May 13th instead. The video game developer said it pushed back the release date "to deliver a more refined experience" and promised to publish an article that provides transparency and insights into its dev process. To deliver a more refined experience, we're sharing that: Wrath of the Druids will now release on May 13 We're working on an article to provide transparency and share insights on our dev process Thanks for your patience. Keep an eye on our social channels for future news!

Considering how long Netflix has been in the streaming business, it has stacked up heaps of data about its viewers, such as their age, gender, location, their taste in media, to name a few. By gathering information across every customer interaction, Netflix can dive right into the minds of its viewers and get an idea of what they might like to watch next even before they finish a show or movie. We have data that suggests there is different viewing behavior depending on the day of the week, the time of day, the device, and sometimes even the location. Netflix has a massive user base of more than 140 million subscribers. Over time, Netflix has deployed several algorithms and mechanisms that make use of this data and generate critical insights that help steer the company in the right direction.

SDL (LSE: SDL), the intelligent language and content company, announces it has entered into a technical partnership with DRUID, specialists in conversational AI, to launch multi-lingual virtual assistants for enterprise organizations that enable real-time communication through chatbots. By integrating SDL Machine Translation with DRUID virtual assistants, companies will be able to conduct chatbot conversations in different languages with employees, customers, partners and suppliers. The solution offers a real-time "interpreter mode" function, which can translate conversations along with "live chat" which can translate into multiple languages in real-time. This supports the need for customers to easily translate entire conversations as well as enable scenarios where an agent – human or virtual – needs to communicate across multiple languages simultaneously. Chatbots are commonly configured to undergo complicated question-and-answering activities in different languages, but language-specific customization can be complex, time-consuming and costly.

On a breezy evening this past weekend, I sat out on my patio, lit a sizable joint, and took little drags from it til the burn line singed my fingertips. When I stood up I was stoned, and I knew it; I rarely smoke pot, so when I do I really feel it. But how high was I, really? I reached for my phone, logged into an app called Druid, and took a five minute test. Your DRUID impairment score is 50.3.