Government
Accuracy is Not Agreement: Expert-Aligned Evaluation of Crash Narrative Classification Models
Bhagat, Sudesh Ramesh, Shihab, Ibne Farabi, Sharma, Anuj
This study investigates the relationship between deep learning (DL) model accuracy and expert agreement in classifying crash narratives. We evaluate five DL models -- including BERT variants, USE, and a zero-shot classifier -- against expert labels and narratives, and extend the analysis to four large language models (LLMs): GPT-4, LLaMA 3, Qwen, and Claude. Our findings reveal an inverse relationship: models with higher technical accuracy often show lower agreement with human experts, while LLMs demonstrate stronger expert alignment despite lower accuracy. We use Cohen's Kappa and Principal Component Analysis (PCA) to quantify and visualize model-expert agreement, and employ SHAP analysis to explain misclassifications. Results show that expert-aligned models rely more on contextual and temporal cues than location-specific keywords. These findings suggest that accuracy alone is insufficient for safety-critical NLP tasks. We argue for incorporating expert agreement into model evaluation frameworks and highlight the potential of LLMs as interpretable tools in crash analysis pipelines.
SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Data
Mishra, Samarth, Saenko, Kate, Saligrama, Venkatesh
Compositionality, or correctly recognizing scenes as compositions of atomic visual concepts, remains difficult for multimodal large language models (MLLMs). Even state of the art MLLMs such as GPT-4o can make mistakes in distinguishing compositions like "dog chasing cat" vs "cat chasing dog". While on Winoground, a benchmark for measuring such reasoning, MLLMs have made significant progress, they are still far from a human's performance. W e show that compositional reasoning in these models can be improved by elucidating such concepts via data, where a model is trained to prefer the correct caption for an image over a close but incorrect one. W e introduce SCRAMBLe: Synthetic Compositional Reasoning Augmentation of MLLMs with Binary preference Learning, an approach for preference tuning open-weight MLLMs on synthetic preference data generated in a fully automated manner from existing image-caption data. SCRAMBLe holistically improves these MLLMs' compositional reasoning capabilities which we can see through significant improvements across multiple vision language composition-ality benchmarks, as well as smaller but significant improvements on general question answering tasks. As a sneak peek, SCRAMBLe tuned Molmo-7B model improves on Winoground from 49.5% to 54.8% (best reported to date), while improving by 1% on more general visual question answering tasks.
Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Study
Holtdirk, Tobias, Assenmacher, Dennis, Bleier, Arnim, Wagner, Claudia
Survey researchers face two key challenges: the rising costs of probability samples and missing data (e.g., non-response or attrition), which can undermine inference and increase the use of convenience samples. Recent work explores using large language models (LLMs) to simulate respondents via persona-based prompts, often without labeled data. We study a more practical setting where partial survey responses exist: we fine-tune LLMs on available data to impute self-reported vote choice under both random and systematic nonresponse, using the German Longitudinal Election Study. We compare zero-shot prompting and supervised fine-tuning against tabular classifiers (e.g., CatBoost) and test how different convenience samples (e.g., students) used for fine-tuning affect generalization. Our results show that when data are missing completely at random, fine-tuned LLMs match tabular classifiers but outperform zero-shot approaches. When only biased convenience samples are available, fine-tuning small (3B to 8B) open-source LLMs can recover both individual-level predictions and population-level distributions more accurately than zero-shot and often better than tabular methods. This suggests fine-tuned LLMs offer a promising strategy for researchers working with non-probability samples or systematic missing-ness, and may enable new survey designs requiring only easily accessible subpopulations.
BOE-XSUM: Extreme Summarization in Clear Language of Spanish Legal Decrees and Notifications
García, Andrés Fernández, de la Rosa, Javier, Gonzalo, Julio, Morante, Roser, Amigó, Enrique, Benito-Santos, Alejandro, Carrillo-de-Albornoz, Jorge, Fresno, Víctor, Ghajari, Adrian, Marco, Guillermo, Plaza, Laura, Salido, Eva Sánchez
The ability to summarize long documents succinctly is increasingly important in daily life due to information overload, yet there is a notable lack of such summaries for Spanish documents in general, and in the legal domain in particular. In this work, we present BOE-XSUM, a curated dataset comprising 3,648 concise, plain-language summaries of documents sourced from Spain's ``Bolet\'ın Oficial del Estado'' (BOE), the State Official Gazette. Each entry in the dataset includes a short summary, the original text, and its document type label. We evaluate the performance of medium-sized large language models (LLMs) fine-tuned on BOE-XSUM, comparing them to general-purpose generative models in a zero-shot setting. Results show that fine-tuned models significantly outperform their non-specialized counterparts. Notably, the best-performing model -- BERTIN GPT-J 6B (32-bit precision) -- achieves a 24\% performance gain over the top zero-shot model, DeepSeek-R1 (accuracies of 41.6\% vs.\ 33.5\%).
Guided Uncertainty Learning Using a Post-Hoc Evidential Meta-Model
Barker, Charmaine, Bethell, Daniel, Gerasimou, Simos
Reliable uncertainty quantification remains a major obstacle to the deployment of deep learning models under distributional shift. Existing post-hoc approaches that retrofit pretrained models either inherit misplaced confidence or merely reshape predictions, without teaching the model when to be uncertain. We introduce GUIDE, a lightweight evidential learning meta-model approach that attaches to a frozen deep learning model and explicitly learns how and when to be uncertain. GUIDE identifies salient internal features via a calibration stage, and then employs these features to construct a noise-driven curriculum that teaches the model how and when to express uncertainty. GUIDE requires no retraining, no architectural modifications, and no manual intermediate-layer selection to the base deep learning model, thus ensuring broad applicability and minimal user intervention. The resulting model avoids distilling overconfidence from the base model, improves out-of-distribution detection by ~77% and adversarial attack detection by ~80%, while preserving in-distribution performance. Across diverse benchmarks, GUIDE consistently outperforms state-of-the-art approaches, evidencing the need for actively guiding uncertainty to close the gap between predictive confidence and reliability.
Mysterious black pyramid UFO seen flying over US in broad daylight
Taylor, your album should be'Life of a Callgirl'. KENNEDY's appalled take on Swift's new record... and its ultra-vivid sex shout outs for Travis the Sasquatch The truth about Keith Urban's guitarist'other woman' Maggie Baugh revealed amid Nicole Kidman divorce How I look like this at 62. I've lost 5 stone fast, 20 years off my biological age and wear size 8... without weight-loss jabs. Hollywood A-listers pay me $50,000 to cure their drug addicted nepo-babies because they can't afford for these secrets to go public Shroud of Turin mystery deepens as surgeon spots hidden detail that points to Jesus' resurrection I'm no longer sleeping with my husband - and never will again, says MOLLY RYDDELL. I love him, but counted down the moments until he climaxed. Then I couldn't bear it any more and the truth spilled out... so many women feel the same Fans erupt at Taylor Swift's'dig' at Travis Kelce's ex Kayla Nicole in wild The Life of a Showgirl track Trump dollar coin design released by Treasury... and it's inspired by an iconic political photo Lori Loughlin's husband Mossimo Giannulli seen with mystery brunette in tiny skirt day after shock split Trump appears alongside Melania at dinner hosted by JD Vance and Usha after'disappearance' rumors Top plastic surgeons reveal secrets behind Taylor Swift's'changing' face: 'It is looking very full' I'm a woman with autism... here are the signs you might be masking, even from yourself Cake-faced 90s sitcom star looks unrecognizable as she ditches the heavy eyeshadow for an LA errand run can you guess who?
Newsom signs AI transparency bill prioritizing safety
Things to Do in L.A. Tap to enable a layout that focuses on the article. Gov. Gavin Newsom holds a news conference at the Google office in San Francisco in August to announce new AI partnerships. This is read by an automated voice. Please report any issues or inconsistencies here . Gov. Gavin Newsom signed legislation Monday requiring AI companies to publicly disclose security protocols and report critical safety incidents.
Exploding number of solar storms feared to spark deadly health emergency for more than half of humanity
The truth about Keith Urban's guitarist'other woman' Maggie Baugh revealed amid Nicole Kidman divorce Top plastic surgeons reveal secrets behind Taylor Swift's'changing' face: 'It is looking very full' Shroud of Turin mystery deepens as surgeon spots hidden detail that points to Jesus' resurrection Hollywood A-listers pay me $50,000 to cure their drug addicted nepo-babies because they can't afford for these secrets to go public Trump dollar coin design released by Treasury... and it's inspired by an iconic political photo I'm no longer sleeping with my husband - and never will again, says MOLLY RYDDELL. I love him, but counted down the moments until he climaxed. Then I couldn't bear it any more and the truth spilled out... so many women feel the same Fans erupt at Taylor Swift's'dig' at Travis Kelce's ex Kayla Nicole in wild The Life of a Showgirl track Taylor, your album should be'Life of a Callgirl'. KENNEDY's appalled take on Swift's new record... and its ultra-vivid sex shout outs for Travis the Sasquatch I was so happy after trying a trendy new cosmetic procedure. But 10 years later I suffered a devastating side effect... the doctor had lied Lori Loughlin's husband Mossimo Giannulli seen with mystery brunette in tiny skirt day after shock split The'middle-class kinks' saving marriages: Wives reveal the eight buzzy sex trends that revived their lagging libidos - including the fantasy husbands are secretly obsessed with I'm a woman with autism... here are the signs you might be masking, even from yourself Cake-faced 90s sitcom star looks unrecognizable as she ditches the heavy eyeshadow for an LA errand run can you guess who?