Goto

Collaborating Authors

 similarity 0



$\texttt{BluePrint}$: A Social Media User Dataset for LLM Persona Evaluation and Training

Bück-Kaeffer, Aurélien, Chooi, Je Qin, Zhao, Dan, Touzel, Maximilian Puelma, Pelrine, Kellin, Godbout, Jean-François, Rabbany, Reihaneh, Yang, Zachary

arXiv.org Artificial Intelligence

Large language models (LLMs) offer promising capabilities for simulating social media dynamics at scale, enabling studies that would be ethically or logistically challenging with human subjects. However, the field lacks standardized data resources for fine-tuning and evaluating LLMs as realistic social media agents. We address this gap by introducing SIMPACT, the SIMulation-oriented Persona and Action Capture Toolkit, a privacy respecting framework for constructing behaviorally-grounded social media datasets suitable for training agent models. We formulate next-action prediction as a task for training and evaluating LLM-based agents and introduce metrics at both the cluster and population levels to assess behavioral fidelity and stylistic realism. As a concrete implementation, we release BluePrint, a large-scale dataset built from public Bluesky data focused on political discourse. BluePrint clusters anonymized users into personas of aggregated behaviours, capturing authentic engagement patterns while safeguarding privacy through pseudonymization and removal of personally identifiable information. The dataset includes a sizable action set of 12 social media interaction types (likes, replies, reposts, etc.), each instance tied to the posting activity preceding it. This supports the development of agents that use context-dependence, not only in the language, but also in the interaction behaviours of social media to model social media users. By standardizing data and evaluation protocols, SIMPACT provides a foundation for advancing rigorous, ethically responsible social media simulations. BluePrint serves as both an evaluation benchmark for political discourse modeling and a template for building domain specific datasets to study challenges such as misinformation and polarization.


Benchmark_Sample_Efficiency_neurips_data

Wenhao Gao

Neural Information Processing Systems

Table 4: We report the mean and standard deviation of AUC Top-10 from 5 independent runs. Figure 9. Though SA_Score is not a great metric, we could see that synthesis-based methods have The diversity is defined as the averaged internal distance within a batch of molecules, measured by Tanimoto similarity. We could see a general trend that the stronger a model is in optimization, the less diverse the results are. In this section, we elaborate the implementation details for each method. To avoid the bias introduced by different dataset, e.g., ZINC, ChemBL, for all the methods, we use ZINC to (i) train/pretrain the model; (ii) provide initial molecule set and (iii) extract vocabulary set.


Automated Journalistic Questions: A New Method for Extracting 5W1H in French

Verhaverbeke, Maxence, Gramaccia, Julie A., Khoury, Richard

arXiv.org Artificial Intelligence

The 5W1H questions -- who, what, when, where, why and how -- are commonly used in journalism to ensure that an article describes events clearly and systematically. Answering them is a crucial prerequisites for tasks such as summarization, clustering, and news aggregation. In this paper, we design the first automated extraction pipeline to get 5W1H information from French news articles. To evaluate the performance of our algorithm, we also create a corpus of 250 Quebec news articles with 5W1H answers marked by four human annotators. Our results demonstrate that our pipeline performs as well in this task as the large language model GPT-4o.


An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation

Gandhi, Shubham, Naik, Atharva, Xie, Yiqing, Rose, Carolyn

arXiv.org Artificial Intelligence

We study cost-efficient collaboration between strong and weak language models for repository-level code generation, where the weak model handles simpler tasks at lower cost, and the most challenging tasks are delegated to the strong model. While many works propose architectures for this task, few analyze performance relative to cost. We evaluate a broad spectrum of collaboration strategies: context-based, pipeline-based, and dynamic, on GitHub issue resolution. Our most effective collaborative strategy achieves equivalent performance to the strong model while reducing the cost by 40%. Based on our findings, we offer actionable guidelines for choosing collaboration strategies under varying budget and performance constraints. Our results show that strong-weak collaboration substantially boosts the weak model's performance at a fraction of the cost, pipeline and context-based methods being most efficient. We release the code for our work at https://github.com/shubhamrgandhi/codegen-strong-weak-collab.


Investigating Literary Motifs in Ancient and Medieval Novels with Large Language Models

Hallenberg, Emelie

arXiv.org Artificial Intelligence

The Greek fictional narratives often termed love novels or romances, ranging from the first century CE to the middle of the 15th century, have long been considered as similar in many ways, not least in the use of particular literary motifs. By applying the use of fine-tuned large language models, this study aims to investigate which motifs exactly that the texts in this corpus have in common, and in which ways they differ from each other. The results show that while some motifs persist throughout the corpus, others fluctuate in frequency, indicating certain trends or external influences. Conclusively, the method proves to adequately extract literary motifs according to a set definition, providing data for both quantitative and qualitative analyses.


GenMol: A Drug Discovery Generalist with Discrete Diffusion

Lee, Seul, Kreis, Karsten, Veccham, Srimukh Prasad, Liu, Meng, Reidenbach, Danny, Peng, Yuxing, Paliwal, Saee, Nie, Weili, Vahdat, Arash

arXiv.org Artificial Intelligence

Drug discovery is a complex process that involves multiple scenarios and stages, such as fragment-constrained molecule generation, hit generation and lead optimization. However, existing molecular generative models can only tackle one or two of these scenarios and lack the flexibility to address various aspects of the drug discovery pipeline. In this paper, we present Generalist Molecular generative model (GenMol), a versatile framework that addresses these limitations by applying discrete diffusion to the Sequential Attachment-based Fragment Embedding (SAFE) molecular representation. GenMol generates SAFE sequences through non-autoregressive bidirectional parallel decoding, thereby allowing utilization of a molecular context that does not rely on the specific token ordering and enhanced computational efficiency. Moreover, under the discrete diffusion framework, we introduce fragment remasking, a strategy that optimizes molecules by replacing fragments with masked tokens and regenerating them, enabling effective exploration of chemical space. GenMol significantly outperforms the previous GPT-based model trained on SAFE representations in de novo generation and fragment-constrained generation, and achieves state-of-the-art performance in goal-directed hit generation and lead optimization. These experimental results demonstrate that GenMol can tackle a wide range of drug discovery tasks, providing a unified and versatile approach for molecular design.


Connecting Large Language Models with Blockchain: Advancing the Evolution of Smart Contracts from Automation to Intelligence

Xian, Youquan, Zeng, Xueying, Xuan, Duancheng, Yang, Danping, Li, Chunpei, Fan, Peng, Liu, Peng

arXiv.org Artificial Intelligence

Blockchain smart contracts have catalyzed the development of decentralized applications across various domains, including decentralized finance. However, due to constraints in computational resources and the prevalence of data silos, current smart contracts face significant challenges in fully leveraging the powerful capabilities of Large Language Models (LLMs) for tasks such as intelligent analysis and reasoning. To address this gap, this paper proposes and implements a universal framework for integrating LLMs with blockchain data, {\sysname}, effectively overcoming the interoperability barriers between blockchain and LLMs. By combining semantic relatedness with truth discovery methods, we introduce an innovative data aggregation approach, {\funcname}, which significantly enhances the accuracy and trustworthiness of data generated by LLMs. To validate the framework's effectiveness, we construct a dataset consisting of three types of questions, capturing Q\&A interactions between 10 oracle nodes and 5 LLM models. Experimental results demonstrate that, even with 40\% malicious nodes, the proposed solution improves data accuracy by an average of 17.74\% compared to the optimal baseline. This research not only provides an innovative solution for the intelligent enhancement of smart contracts but also highlights the potential for deep integration between LLMs and blockchain technology, paving the way for more intelligent and complex applications of smart contracts in the future.


Robust Table Integration in Data Lakes

Ji, Daomin, Luo, Hui, Bao, Zhifeng, Culpepper, Shane

arXiv.org Artificial Intelligence

In this paper, we investigate the challenge of integrating tables from data lakes, focusing on three core tasks: 1) pairwise integrability judgment, which determines whether a tuple pair in a table is integrable, accounting for any occurrences of semantic equivalence or typographical errors; 2) integrable set discovery, which aims to identify all integrable sets in a table based on pairwise integrability judgments established in the first task; 3) multi-tuple conflict resolution, which resolves conflicts among multiple tuples during integration. We train a binary classifier to address the task of pairwise integrability judgment. Given the scarcity of labeled data, we propose a self-supervised adversarial contrastive learning algorithm to perform classification, which incorporates data augmentation methods and adversarial examples to autonomously generate new training data. Upon the output of pairwise integrability judgment, each integrable set is considered as a community, a densely connected sub-graph where nodes and edges correspond to tuples in the table and their pairwise integrability, respectively. We proceed to investigate various community detection algorithms to address the integrable set discovery objective. Moving forward to tackle multi-tuple conflict resolution, we introduce an novel in-context learning methodology. This approach capitalizes on the knowledge embedded within pretrained large language models to effectively resolve conflicts that arise when integrating multiple tuples. Notably, our method minimizes the need for annotated data. Since no suitable test collections are available for our tasks, we develop our own benchmarks using two real-word dataset repositories: Real and Join. We conduct extensive experiments on these benchmarks to validate the robustness and applicability of our methodologies in the context of integrating tables within data lakes.


Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models

Bavaresco, A., Testoni, A., Fernández, R.

arXiv.org Artificial Intelligence

Image-based advertisements are complex multimodal stimuli that often contain unusual visual elements and figurative language. Previous research on automatic ad understanding has reported impressive zero-shot accuracy of contrastive vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we examine the original task setup and show that contrastive VLMs can solve it by exploiting grounding heuristics. To control for this confound, we introduce TRADE, a new evaluation test set with adversarial grounded explanations. While these explanations look implausible to humans, we show that they "fool" four different contrastive VLMs. Our findings highlight the need for an improved operationalisation of automatic ad understanding that truly evaluates VLMs' multimodal reasoning abilities. We make our code and TRADE available at https://github.com/dmg-illc/trade .