As enthusiasm for scaling computation (data and parameters) in the pretraining era gradually diminished, test-time scaling (TTS), also referred to as ``test-time computing'' has emerged as a prominent research focus. Recent studies demonstrate that TTS can further elicit the problem-solving capabilities of large language models (LLMs), enabling significant breakthroughs not only in specialized reasoning tasks, such as mathematics and coding, but also in general tasks like open-ended Q&A. However, despite the explosion of recent efforts in this area, there remains an urgent need for a comprehensive survey offering a systemic understanding. To fill this gap, we propose a unified, multidimensional framework structured along four core dimensions of TTS research: what to scale, how to scale, where to scale, and how well to scale. Building upon this taxonomy, we conduct an extensive review of methods, application scenarios, and assessment aspects, and present an organized decomposition that highlights the unique functional roles of individual techniques within the broader TTS landscape. From this analysis, we distill the major developmental trajectories of TTS to date and offer hands-on guidelines for practical deployment. Furthermore, we identify several open challenges and offer insights into promising future directions, including further scaling, clarifying the functional essence of techniques, generalizing to more tasks, and more attributions. Our repository is available on https://github.com/testtimescaling/testtimescaling.github.io/

In the update on Monday, Mr Altman said the non-profit would continue to control OpenAI, receiving a "big", yet-to-be determined stake in OpenAI's business arm, which would give it access to money to put towards its own goals. He said the new plan would still allow the organisation to stop operating under its current complex governance structure, which had capped its profits. That was seen as a sticking point for investors, including Microsoft, that hindered the firm's ability to raise money. "We are moving to a normal capital structure where everyone has stock," he wrote in a letter to staff that was shared on the OpenAI website. "This is not a sale, but a change of structure to something simpler."

OpenAI on Monday announced a proposed restructuring that would give its nonprofit arm ongoing control of ChatGPT and the rest of the startup's AI products. The move is a reversal of an earlier announcement which called for the nonprofit to relinquish its authority to a newly created public-benefit corporation. The proposed company structure has to be approved by the attorney general offices in California and Delaware by early next year. Up to 30 billion in funding from SoftBank and other investors is contingent on this approval. That money is crucial for OpenAI to maintain its position as a leader in generative AI and give higher returns to investors.

OpenAI has reversed course in the process of transforming into a for-profit entity, announcing on Monday that its non-profit arm would continue to control the business that makes ChatGPT and other artificial intelligence (AI) products. Previously, the company had sought more independence for its for-profit division. "We made the decision for the nonprofit to stay in control after hearing from civic leaders and having discussions with the offices of the Attorneys General of California and Delaware," said CEO Sam Altman in a letter to employees. Altman and the chair of OpenAI's non-profit board, Bret Taylor, said the board made the choice for the non-profit to retain control of OpenAI. A press release from the company said that the for-profit portion of the company, through which Altman has been able to raise billions to fund OpenAI's work, would transition to a public benefit corporation, a mission-driven designation for a corporate structure that is still aimed at profit but also "has to consider the interests of both shareholders and the mission".

Thank you to everyone who attended our most recent AI Unlocked webinar. I really enjoyed our lively discussion about ChatGPT's software features and wish I could have answered even more of your questions about using generative AI tools. I really enjoyed the questions about what ChatGPT can do beyond just chatting. Image search is a feature I use often, and here are my first impressions of the tool that I recorded back in September 2023 when it first dropped. I use ChatGPT's image search tool nowadays by snapping a picture with my phone when I don't recognize something.

Understanding and resolving temporal references is essential in Natural Language Understanding as we often refer to the past or future in daily communication. Although existing benchmarks address a system's ability to reason about and resolve temporal references, systematic evaluation of specific temporal references remains limited. Towards closing this gap, we introduce TRAVELER, a novel synthetic benchmark dataset that follows a Question Answering paradigm and consists of questions involving temporal references with the corresponding correct answers. TRAVELER assesses models' abilities to resolve explicit, implicit relative to speech time, and vague temporal references. Beyond investigating the performance of state-of-the-art LLMs depending on the type of temporal reference, our benchmark also allows evaluation of performance in relation to the length of the set of events. For the category of vague temporal references, ground-truth answers were established via human surveys on Prolific, following a procedure similar to the one from Kenneweg et al. To demonstrate the benchmark's applicability, we evaluate four state-of-the-art LLMs using a question-answering task encompassing 3,300 questions. Our findings show that while the benchmarked LLMs can answer questions over event sets with a handful of events and explicit temporal references successfully, performance clearly deteriorates with larger event set length and when temporal references get less explicit. Notably, the vague question category exhibits the lowest performance across all models. The benchmark is publicly available at: https://gitlab.ub.uni-bielefeld.de/s.kenneweg/TRAVELER

Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents D ocument R etrieval-A ugmented F ine-T uning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) fo r safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by intro ducing a novel fine-tuning framework that accommodates our dual-re trieval architecture, which simultaneously accesses both softwar e documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodolog y that incorporates variable numbers of relevant documents with m eaning-ful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT -4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative impr ovements in evidence handling, response structure, and domain-spec ific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transpar ency and evidence-based reasoning essential in regulatory domains .

As Natural Language Processing (NLP) models continue to evolve and become integral to high-stakes applications, ensuring their interpretability remains a critical challenge. Given the growing variety of explainability methods and diverse stakeholder requirements, frameworks that help stakeholders select appropriate explanations tailored to their specific use cases are increasingly important. To address this need, we introduce EvalxNLP, a Python framework for benchmarking state-of-the-art feature attribution methods for transformer-based NLP models. EvalxNLP integrates eight widely recognized explainability techniques from the Explainable AI (XAI) literature, enabling users to generate and evaluate explanations based on key properties such as faithfulness, plausibility, and complexity. Our framework also provides interactive, LLM-based textual explanations, facilitating user understanding of the generated explanations and evaluation outcomes. Human evaluation results indicate high user satisfaction with EvalxNLP, suggesting it is a promising framework for benchmarking explanation methods across diverse user groups. By offering a user-friendly and extensible platform, EvalxNLP aims at democratizing explainability tools and supporting the systematic comparison and advancement of XAI techniques in NLP.

While research on applications and evaluations of explanation methods continues to expand, fairness of the explanation methods concerning disparities in their performance across subgroups remains an often overlooked aspect. In this paper, we address this gap by showing that, across three tasks and five language models, widely used post-hoc feature attribution methods exhibit significant gender disparity with respect to their faithfulness, robustness, and complexity. These disparities persist even when the models are pre-trained or fine-tuned on particularly unbiased datasets, indicating that the disparities we observe are not merely consequences of biased training data. Our results highlight the importance of addressing disparities in explanations when developing and applying explainability methods, as these can lead to biased outcomes against certain subgroups, with particularly critical implications in high-stakes contexts. Furthermore, our findings underscore the importance of incorporating the fairness of explanations, alongside overall model fairness and explainability, as a requirement in regulatory frameworks.

As large language models (LLMs) continue to evolve, it is critical to assess the security threats and vulnerabilities that may arise both during their training phase and after models have been deployed. This survey seeks to define and categorize the various attacks targeting LLMs, distinguishing between those that occur during the training phase and those that affect already trained models. A thorough analysis of these attacks is presented, alongside an exploration of defense mechanisms designed to mitigate such threats. Defenses are classified into two primary categories: prevention-based and detection-based defenses. Furthermore, our survey summarizes possible attacks and their corresponding defense strategies. It also provides an evaluation of the effectiveness of the known defense mechanisms for the different security threats. Our survey aims to offer a structured framework for securing LLMs, while also identifying areas that require further research to improve and strengthen defenses against emerging security challenges.