Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the base parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, deriving that the base of RoPE bounds context length: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling.

To benchmark a model's long-form factuality in open domains, we first use GPT -4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE).

Instead, we propose Meta Linear Attention (MetaLA) as a solution that satisfies these conditions.

Depending on foreign-made open models is both a supply chain risk and an innovation problem, experts say. Since 2022, America has had a solid lead in artificial intelligence thanks to advanced models from high-flying companies like OpenAI, Google DeepMind, Anthropic, and xAI. A growing number of experts, however, worry that the US is starting to fall behind when it comes to minting open-weight AI models that can be downloaded, adapted, and run locally. Open models from Chinese companies like Kimi, Z.ai, Alibaba, and DeepSeek are now rapidly gaining popularity among researchers and engineers worldwide, leaving the US as a laggard in an increasingly vital area of AI innovation. "The US needs open models to cement its lead at every level of the AI stack," Nathan Lambert, founder of the ATOM (American Truly Open Models) Project, tells WIRED.

Further analysis substantiates our hypothesis that our improvement can be attributed to reduced overfitting to instruction tuning datasets. It is worth noting that we are not proposing IM as a replacement for the current instruction tuning process. Instead, our work aims to provide practical guidance for instruction tuning LMs, especially in low-resource scenarios.

Knowledge editing technology has received widespread attention for low-cost updates of incorrect or outdated knowledge in large-scale language models. However, recent research has found that edited models often exhibit varying degrees of performance degradation.

Chen et al., 2021], and come up with new problems that drive innovation in art and science [Gromov,

The framework enhances Large Language Models' (LLMs) embodied reasoning capabilities by efficiently exploring