We present FLUKE (Framework for LingUistically-driven and tasK-agnostic robustness Evaluation), a framework for assessing model robustness through systematic minimal variations of test data. FLUKE introduces controlled variations across linguistic levels -- from orthography to dialect and style -- and leverages large language models (LLMs) with human validation to generate modifications. We demonstrate FLUKE's utility by evaluating both fine-tuned models and LLMs across six diverse NLP tasks (four classification and two generation tasks), and reveal that (1) the impact of linguistic variations is highly task-dependent, with some tests being critical for certain tasks but irrelevant for others; (2) LLMs still exhibit significant brittleness to certain linguistic variations, with reasoning LLMs surprisingly showing less robustness on some tasks compared to base models; (3) models are overall more brittle to natural, fluent modifications such as syntax or style changes (and especially to negation), compared to corruption-style tests such as letter flipping; (4) the ability of a model to use a linguistic feature in generation does not correlate to its robustness to this feature on downstream tasks. These findings highlight the importance of systematic robustness testing for understanding model behaviors.

Breakthroughs, discoveries, and DIY tips sent every weekday. It is in these waters that marine mammal ecologist Ari Friedlaender shuts off the inflatable boat's engine and waits. This is the edge of the world--remote, hostile, and stunningly alive. Beneath the hull, the dark sea churns with wonder abound. A humpback whale (Megaptera novaeangliae) emerges, slow, deliberate, and gentle in its curious demeanor, casting a ripple across the surface.

Since its inception in 2016, Federated Learning (FL) has been gaining tremendous popularity in the machine learning community. Several frameworks have been proposed to facilitate the development of FL algorithms, but researchers often resort to implementing their algorithms from scratch, including all baselines and experiments. This is because existing frameworks are not flexible enough to support their needs or the learning curve to extend them is too steep. In this paper, we present \fluke, a Python package designed to simplify the development of new FL algorithms. fluke is specifically designed for prototyping purposes and is meant for researchers or practitioners focusing on the learning components of a federated system. fluke is open-source, and it can be either used out of the box or extended with new algorithms with minimal overhead.

Using machine learning, researchers calculated that in roughly 35 years its ecological range will move northward, bringing hay fever to regions it has …

Last June, a simple question changed my life: "Hey Bryn, what do you know about whales?" Of course like most people, my answer was "Not much," but that marked the beginning of an important project to help track the health of whale populations by using machine learning to analyze video from drones. Parley for the Oceans introduced me and my colleagues Ted Willke and Javier Turek to Dr. Iain Kerr of Ocean Alliance, and we started talking about how machine learning could help make marine biologists' lives easier and help protect the whales, using the video from Dr. Kerr's SnotBot drones. Dr. Kerr started the SnotBot program because in the not-so-distant past, when people wanted to understand the health of a whale, or get its DNA, the only way to do this was to shoot it with a crossbow with a specially prepared bolt with a string on it, that would only go a couple of inches into the (remember, bus-size) body. The bolt would then be reeled in and the sample could be extracted from it.