Large language models (LLMs) have achieved impressive performance across various natural language benchmarks, prompting a continual need to curate more difficult datasets for larger LLMs, which is costly and time-consuming. In this paper, we propose to automate dataset updating and provide systematical analysis regarding its effectiveness in dealing with benchmark leakage issue, difficulty control, and stability. Thus, once current benchmark has been mastered or leaked, we can update it for timely and reliable evaluation. There are two updating strategies: 1) mimicking strategy to generate similar samples based on original data, preserving stylistic and contextual essence, and 2) extending strategy that further expands existing samples at varying cognitive levels by adapting Bloom's taxonomy of educational objectives. Extensive experiments on updated MMLU and BIG-Bench demonstrate the stability of the proposed strategies and find that the mimicking strategy can effectively alleviate issues of overestimation from benchmark leakage. In cases where the efficient mimicking strategy fails, our extending strategy still shows promising results. Additionally, by controlling the difficulty, we can better discern the models' performance and enable fine-grained analysis -- neither too difficult nor too easy an exam can fairly judge students' learning status. To the best of our knowledge, we are the first to automate updating benchmarks for reliable and timely evaluation.

Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases.

THE LOST WEDDING PHOTOS: See JFK Jr and Carolyn Bessette at their secret nuptials... and read every intimate detail of ultra-private ceremony Tulsi Gabbard lets Iran nuke bombshell slip as Senate hearing spirals for Trump's embattled spy chief Candace Owens's sickening low-blow at Karoline Leavitt as Iran war sparks wild attacks Lunatic Megyn Kelly is FINALLY ruined! Her appalling X-rated smear of my friend proves it... but now I know her truly disturbing plan: JOSH HAMMER Inside the epidemic of midlife women who are repulsed by their husbands, the age and'vital statistics' that make men most at risk - and the telltale signs YOUR marriage is about to die: Special report by SADIE NICHOLAS Meghan gives glimpse of'mama's little helpers' Archie and Lilibet in'behind the scenes' video of her latest As Ever launch Shameful hypocrisy of NASCAR star Daniel Suarez's nepo-baby wife: 'Victim' mask slips as she ignites new Las Vegas drama... and dark family past rears its ugly head Princess Kate dons her favourite tiara and the late Queen's earrings as she arrives at King's banquet for the Nigerian President in country's first state visit in almost 40 years Everything JFK Jr told friends about his love affair with'sexual dynamo' Madonna... her unprintable pillow talk... and his perverse incest request that she couldn't go through with Site of'Jesus' crucifixion' forced to shut for Holy Week in unprecedented move tied to biblical prophecies of the Antichrist Ugly new Nicole Kidman and Keith Urban divorce fight ERUPTS: Her friends share humiliating details of'midlife crisis'... and reveal brutal REAL reason daughter Sunday Rose'snubbed' him Outrage after Seattle museum vandal destroys $250,000 of famous Dale Chihuly glass at city's museum dedicated to him Amanda Bynes, 39, 'is now a size 4 after losing 35lb' thanks to weight-loss medication... after hitting 180lb Chilling unclassified threat report reveals the'most likely' terror attack scenario on US soil Three's Company bombshell Jenilee Harrison who was also on Dallas and The Love Boat still looks great at 67, see her now The discovery of a Roman artifact in the Americas has sparked a debate about who truly discovered the New World. While Christopher Columbus is hailed as the first in 1492, archaeologists uncovered a small terracotta head of a bearded man carved with distinctive European features tucked inside a Mexican tomb. The artifact, known as the Tecaxic-Calixtlahuaca Head, was discovered in 1933 inside a sealed pre-Hispanic burial beneath multiple intact layers, indicating it had not been disturbed after its placement. Experts say its facial features, beard style and craftsmanship bear a striking resemblance to objects from the ancient Mediterranean rather than indigenous Mesoamerican traditions.

It will likely go through several more iterations before completion. The White House has been promising a set of national rules to guide artificial intelligence since late last year, and today Sen. Marsha Blackburn (R-Tenn.) The senator shared a discussion draft for codifying the executive order signed by President Donald Trump in December calling for an AI bill. Her stated goal is a policy that protects children, creators, conservatives and communities from harm. Blackburn has called for tougher policies for AI safety, and one of the core messages in this discussion draft is that it places a duty of care on AI developers in the design, development and operation of AI platforms to prevent and mitigate foreseeable harm to users.

The company had originally planned to announce global availability. A few hours after this story was published, Google reached out to retract the news. Given that the company says it is testing in more markets, it seems entirely possible that the global Search Live release will happen sooner than later. After rolling out Search Live to all US Google app users last September, Google is now bringing the feature to every place where it offers its AI Mode chatbot. Search Live, if you need a reminder, allows you to point your phone's camera at an object or scene and ask questions about what you see in front of you.

The Federated Averaging (FedAvg) algorithm, which consists of alternating between a few local stochastic gradient updates at client nodes, followed by a model averaging update at the server, is perhaps the most commonly used method in Federated Learning. Notwithstanding its simplicity, several empirical studies have illustrated that the model output by FedAvg leads to a model that generalizes well to new unseen tasks after a few fine-tuning steps. This surprising performance of such a simple method, however, is not fully understood from a theoretical point of view. In this paper, we formally investigate this phenomenon in the multi-task linear regression setting. We show that the reason behind the generalizability of the FedAvg output is FedAvg's power in learning the common data representation among the clients' tasks, by leveraging the diversity among client data distributions via multiple local updates between communication rounds. We formally establish the iteration complexity required by the clients for proving such result in the setting where the underlying shared representation is a linear map. To the best of our knowledge, this is the first result showing that FedAvg learns an expressive representation in any setting. Moreover, we show that multiple local updates between communication rounds are necessary for representation learning, as distributed gradient methods that make only one local update between rounds provably cannot recover the ground-truth representation in the linear setting, and empirically yield neural network representations that generalize drastically worse to new clients than those learned by FedAvg trained on heterogeneous image classification datasets.

Vision-language models (VLMs) have made significant progress in recent visual-question-answering (VQA) benchmarks that evaluate complex visio-linguistic reasoning. However, are these models truly effective? In this work, we show that VLMs still struggle with natural images and questions that humans can easily answer, which we term $\textbf{natural adversarial samples}$. We also find it surprisingly easy to generate these VQA samples from natural image-text corpora using off-the-shelf models like CLIP and ChatGPT. We propose a semi-automated approach to collect a new benchmark, ${\bf NaturalBench}$, for reliably evaluating VLMs with 10,000 human-verified VQA samples.

Recurrent Neural Networks (RNNs) are used to learn representations in partially observable environments. For agents that learn online and continually interact with the environment, it is desirable to train RNNs with real-time recurrent learning (RTRL); unfortunately, RTRL is prohibitively expensive for standard RNNs. A promising direction is to use linear recurrent architectures (LRUs), where dense recurrent weights are replaced with a complex-valued diagonal, making RTRL efficient. In this work, we build on these insights to provide a lightweight but effective approach for training RNNs in online RL. We introduce Recurrent Trace Units (RTUs), a small modification on LRUs that we nonetheless find to have significant performance benefits over LRUs when trained with RTRL. We find RTUs significantly outperform GRUs and Transformers across several partially observable environments while using significantly less computation.

The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models. Most available corpora have sufficient data only for languages with large dominant communities. However, there is no corpus available that (i) covers a wide range of minority languages; (ii) is generated by an open-source reproducible pipeline; and (iii) is rigorously cleaned from noise, making it trustworthy to use. We present GlotCC, a clean, document-level, 2TB general domain corpus derived from CommonCrawl, covering more than 1000 languages.

Distributed learning is essential for training large-scale deep models.Asynchronous SGD (ASGD) and its variants are commonly used distributed learning methods, particularly in scenarios where the computing capabilities of workers in the cluster are heterogeneous.Momentum has been acknowledged for its benefits in both optimization and generalization in deep model training. However, existing works have found that naively incorporating momentum into ASGD can impede the convergence.In this paper, we propose a novel method called ordered momentum (OrMo) for ASGD. In OrMo, momentum is incorporated into ASGD by organizing the gradients in order based on their iteration indexes. We theoretically prove the convergence of OrMo with both constant and delay-adaptive learning rates for non-convex problems. To the best of our knowledge, this is the first work to establish the convergence analysis of ASGD with momentum without dependence on the maximum delay. Empirical results demonstrate that OrMo can achieve better convergence performance compared with ASGD and other asynchronous methods with momentum.