How can large language models (LLMs) serve users with varying preferences that may conflict across cultural, political, or other dimensions? To advance this challenge, this paper establishes four key results. First, we demonstrate, through a large-scale multilingual human study with representative samples from five countries (N=15,000), that humans exhibit significantly more variation in preferences than the responses of 21 state-of-the-art LLMs. Second, we show that existing methods for preference dataset collection are insufficient for learning the diversity of human preferences even along two of the most salient dimensions of variability in global values, due to the underlying homogeneity of candidate responses. Third, we argue that this motivates the need for negatively-correlated sampling when generating candidate sets, and we show that simple prompt-based techniques for doing so significantly enhance the performance of alignment methods in learning heterogeneous preferences. Fourth, based on this novel candidate sampling approach, we collect and open-source Community Alignment, the largest and most representative multilingual and multi-turn preference dataset to date, featuring almost 200,000 comparisons from annotators spanning five countries. We hope that the Community Alignment dataset will be a valuable resource for improving the effectiveness of LLMs for a diverse global population.

Open AI CEO Sam Altman speaks during a summit on June 2, 2025 in San Francisco, California. Open AI CEO Sam Altman speaks during a summit on June 2, 2025 in San Francisco, California. Founded in 2015 as a nonprofit, rather than a for-profit company, it promised to develop AI "in the way that is most likely to benefit humanity." With billions of dollars in investments from Microsoft, Japanese bank SoftBank, and chipmaker Nvidia, however, OpenAI has proposed changing its corporate structure to give investors more control over its technology. Critics of the change include cofounder-turned-competitor, Elon Musk, and nonprofits concerned about OpenAI's adherence to its mission.

Once in a while, LLMs turn evil--and no one quite knows why. The AI company Anthropic has made a rigorous effort to build a large language model with positive human values. The $183 billion company's flagship product is Claude, and much of the time, its engineers say, Claude is a model citizen. Its standard persona is warm and earnest. When users tell Claude to "answer like I'm a fourth grader" or "you have a PhD in archeology," it gamely plays along. It makes threats and then carries them out. And the frustrating part--true of all LLMs--is that no one knows exactly why. Consider a recent stress test that Anthropic's safety engineers ran on Claude. In their fictional scenario, the model was to take on the role of Alex, an AI belonging to the Summit Bridge corporation.

Erotic imagery and curiosity often arise in intimate relationships, especially when there's safety, play, and mutual recognition. It doesn't mean you've done anything "wrong." On the contrary, it shows that your imagination is alive and searching for ways to bridge the gap between closeness and distance, fantasy and reality. You offer me something charged, even a bit embarrassing, and you're watching--will I crumble?

Ed Zitron Gets Paid to Love AI. He's one of the loudest voices of the AI haters--even as he does PR for AI companies. Either way, Ed Zitron has your attention. In his day job, Ed Zitron runs a boutique public relations firm called EZPR. This might surprise anyone who has come to know Zitron through his podcast or his social media or the newsletter in which he writes two-fisted stuff like "Sam Altman is full of shit and "Mark Zuckerberg is a putrid ghoul." Flacks, as a rule, tend not to talk like this. Flacks send prim, throat-clearing emails to media people who do, on rare occasions, talk like this. Flacks want to touch base, hop on the phone, clear up a few things about the allegation that their CEO is a "chunderfuck." And that really is one of the things with guys like Sam Altman and Dario Amodei from Anthropic," Zitron was saying over burgers on a fine Manhattan afternoon in September. "I work with founders all the time. I'm a founder myself, I guess--I don't like the title. But when you are a person that has to make more money than you lose, otherwise you lose your business, and you see these chunderfucks burning 5, 10 billion dollars in a year--and everyone's celebrating them? We were talking about whether any of Zitron's ranting about the AI industry had cost him business on the PR side of the ledger. There was the one client who felt Zitron was being a little mean toward Altman, the CEO of OpenAI and the biggest chunderfuck of all, as far as Zitron is concerned. Founding a company is hard, the client said. "I said, 'I appreciate the comment, but, like, this isn't about you,'" Zitron told me. "His company is burning billions of dollars.

In recent times, there have been two techno-religious awakenings. To be human is to yearn for a Sky Daddy. Something that explains the unexplainable, someone to blame. No wonder, then, that in the ZIRP-fueled 2010s, when a new gospel of creation was being spread, some people started to see technology as a kind of religion. Startup founders and CEOs became messianic figures.

Reinforcement learning (RL) has become a predominant technique to align language models (LMs) with human preferences or promote outputs which are deemed to be desirable by a given reward function. Standard RL approaches optimize average reward, while methods explicitly focused on reducing the probability of undesired outputs typically come at a cost to average-case performance. To improve this tradeoff, we introduce RePULSe, a new training method that augments the standard RL loss with an additional loss that uses learned proposals to guide sampling low-reward outputs, and then reduces those outputs' probability. We run experiments demonstrating that RePULSe produces a better tradeoff of expected reward versus the probability of undesired outputs and is more adversarially robust, compared to standard RL alignment approaches and alternatives.

Aligning large language models with human feedback at inference time has received increasing attention due to its flexibility. Existing methods rely on generating multiple responses from the base policy for search using a reward model, which can be considered as searching in a discrete response space. However, these methods struggle to explore informative candidates when the base policy is weak or the candidate set is small, resulting in limited effectiveness. In this paper, to address this problem, we propose Simple Energy Adaptation ($\textbf{SEA}$), a simple yet effective algorithm for inference-time alignment. In contrast to expensive search over the discrete space, SEA directly adapts original responses from the base policy toward the optimal one via gradient-based sampling in continuous latent space. Specifically, SEA formulates inference as an iterative optimization procedure on an energy function over actions in the continuous space defined by the optimal policy, enabling simple and effective alignment. For instance, despite its simplicity, SEA outperforms the second-best baseline with a relative improvement of up to $ \textbf{77.51%}$ on AdvBench and $\textbf{16.36%}$ on MATH. Our code is publicly available at https://github.com/yuanyige/sea

Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM backbones. Meanwhile, the increasing availability of high-quality synthetic instruction-following datasets raises the question: can simpler, reference-based metrics serve as viable alternatives to reward models during RL-based alignment? In this paper, we show first that BLEU, a basic string-matching metric, surprisingly matches strong reward models in agreement with human preferences on general instruction-following datasets. Based on this insight, we develop BLEUBERI, a method that first identifies challenging instructions and then applies Group Relative Policy Optimization (GRPO) using BLEU directly as the reward function. We demonstrate that BLEUBERI-trained models are competitive with models trained via reward model-guided RL across four challenging instruction-following benchmarks and three different base language models. A human evaluation further supports that the quality of BLEUBERI model outputs is on par with those from reward model-aligned models. Moreover, BLEUBERI models generate outputs that are more factually grounded than competing methods. Overall, we show that given access to high-quality reference outputs (easily obtained via existing instruction-following datasets or synthetic data generation), string matching-based metrics are cheap yet effective proxies for reward models during alignment. We release our code and data at https://github.com/lilakk/BLEUBERI.

AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose "deep research" systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they (1) fail to provide holistic, product-informed measures of real-world use cases such as science research; (2) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (3) do not account for confounding variables such as model cost and tool access; (4) do not provide standardized interfaces for quick agent prototyping and evaluation; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides the first holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.