This work presents BAdam, an optimization method that leverages the block coordinate descent framework with Adam as the inner solver. BAdam offers a memory efficient approach to the full parameter finetuning of large language models. We conduct theoretical convergence analysis for BAdam in the deterministic case. Experimentally, we apply BAdam to instruction-tune the Llama 2-7B and Llama 3-8B models using a single RTX3090-24GB GPU. The results confirm BAdam's efficiency in terms of memory and running time. Additionally, the convergence verification indicates that BAdam exhibits superior convergence behavior compared to LoRA. Furthermore, the downstream performance evaluation using the MT-bench shows that BAdam modestly surpasses LoRA and more substantially outperforms LOMO. Finally, we compare BAdam with Adam on a medium-sized task, i.e., finetuning RoBERTa-large on the SuperGLUE benchmark. The results demonstrate that BAdam is capable of narrowing the performance gap with Adam more effectively than LoRA. Our code is available at https://github.com/Ledzy/BAdam.

Public data ecosystems (PDEs) represent complex socio-technical systems crucial for optimizing data use in the public sector and outside it. Recognizing their multifaceted nature, previous research pro-posed a six-generation Evolutionary Model of Public Data Ecosystems (EMPDE). Designed as a result of a systematic literature review on the topic spanning three decade, this model, while theoretically robust, necessitates empirical validation to enhance its practical applicability. This study addresses this gap by validating the theoretical model through a real-life examination in five European countries - Latvia, Serbia, Czech Republic, Spain, and Poland. This empirical validation provides insights into PDEs dynamics and variations of implementations across contexts, particularly focusing on the 6th generation of forward-looking PDE generation named "Intelligent Public Data Generation" that represents a paradigm shift driven by emerging technologies such as cloud computing, Artificial Intelligence, Natural Language Processing tools, Generative AI, and Large Language Models (LLM) with potential to contribute to both automation and augmentation of business processes within these ecosystems. By transcending their traditional status as a mere component, evolving into both an actor and a stakeholder simultaneously, these technologies catalyze innovation and progress, enhancing PDE management strategies to align with societal, regulatory, and technical imperatives in the digital era.

In applications such as personal assistants, large language models (LLMs) must consider the user's personal information and preferences. However, LLMs lack the inherent ability to learn from user interactions. This paper explores capturing personal information from user prompts using ontology and knowledge-graph approaches. We use a subset of the KNOW ontology, which models personal information, to train the language model on these concepts. We then evaluate the success of knowledge capture using a specially constructed dataset.

Humans are efficient language learners and inherently social creatures. Our language development is largely shaped by our social interactions, for example, the demonstration and feedback from caregivers. Contrary to human language learning, recent advancements in large language models have primarily adopted a non-interactive training paradigm, and refined pre-trained models through feedback afterward. In this work, we aim to examine how corrective feedback from interactions influences neural language acquisition from the ground up through systematically controlled experiments, assessing whether it contributes to learning efficiency in language models. We introduce a trial-and-demonstration (TnD) learning framework that incorporates three components: student trials, teacher demonstrations, and a reward conditioned on language competence at various developmental stages. Our experiments reveal that the TnD approach accelerates word acquisition for student models of equal and smaller numbers of parameters, and we highlight the significance of both trials and demonstrations. We further show that the teacher's choices of words influence students' word-specific learning efficiency, and a practice-makes-perfect effect is evident by a strong correlation between the frequency of words in trials and their respective learning curves. Our findings suggest that interactive language learning, with teacher demonstrations and student trials, can facilitate efficient word learning in language models.

Training reinforcement learning-based recommender systems are often hindered by the lack of dynamic and realistic user interactions. Lusifer, a novel environment leveraging Large Language Models (LLMs), addresses this limitation by generating simulated user feedback. It synthesizes user profiles and interaction histories to simulate responses and behaviors toward recommended items. In addition, user profiles are updated after each rating to reflect evolving user characteristics. Using the MovieLens100K dataset as proof of concept, Lusifer demonstrates accurate emulation of user behavior and preferences. This paper presents Lusifer's operational pipeline, including prompt generation and iterative user profile updates. While validating Lusifer's ability to produce realistic dynamic feedback, future research could utilize this environment to train reinforcement learning systems, offering a scalable and adjustable framework for user simulation in online recommender systems.

Psychology, with a focus on psychometry, heavily relies on statistical models of analysis to create a cohesive understanding of personality and preferences of individuals. One of the fields that showed a proper evolution over time is the psychology of personality. Statistics-wise, personality can be modeled using the Five Factor Model (FFM), this model being the most scientifically validated personality model to date. It consists of five personality traits, each divided into 6 facets, usually. These traits can be memorized using the acronym OCEAN: openness to experience (O), conscienciousness (C), extraversion (E), agreeableness (A) and neuroticism (N). The traits are not correlated with one another, as evidence suggests. However, these 5 personality traits can be mapped into two metatraits: plasticity and stability.

Safety and trustworthiness are indispensable requirements for real-world applications of AI systems using large language models (LLMs). This paper formulates human value alignment as an optimization problem of the language model policy to maximize reward under a safety constraint, and then proposes an algorithm, Stepwise Alignment for Constrained Policy Optimization (SACPO). One key idea behind SACPO, supported by theory, is that the optimal policy incorporating reward and safety can be directly obtained from a reward-aligned policy. Building on this key idea, SACPO aligns LLMs step-wise with each metric while leveraging simple yet powerful alignment algorithms such as direct preference optimization (DPO). SACPO offers several advantages, including simplicity, stability, computational efficiency, and flexibility of algorithms and datasets. Under mild assumptions, our theoretical analysis provides the upper bounds on optimality and safety constraint violation. Our experimental results show that SACPO can fine-tune Alpaca-7B better than the state-of-the-art method in terms of both helpfulness and harmlessness.

For rapid development in pretraining, we establish a stable and sensitive ablation environment to evaluate and compare experiment runs with minimal model size. We also mainly emphasize data during alignment, where we strive to balance quantity and quality with filtering and reformatting. With tailored data, 360Zhinao-7B's context window is easily extended to 32K and 360K. RMs and RLHF are trained following SFT and credibly applied to specific tasks. All together these contributions lead to 360Zhinao-7B's competitive performance among models of similar size. In recent years, the field of natural language processing (NLP) has witnessed a profound transformation, fueled by the advent of large language models (LLMs) (Bubeck et al., 2023; Touvron et al., 2023a; OpenAI, 2023), which have emerged as a cornerstone to revolutionize the way we understand and generate human language. LLMs represent a new paradigm in artificial intelligence (AI) research, characterized by their immense scale, complexity, and versatility (Zhao et al., 2023). Those models, typically built upon advanced neural network architectures like Transformers, are trained on vast amounts of text data, encompassing billions or even trillions of words. The extensive training endows LLMs with a deep understanding of linguistic structures, nuances, and context, enabling them to generate human-like text and perform a myriad of NLP tasks with unprecedented accuracy and fluency (Yang et al., 2024). Despite the impressive capabilities of LLMs, training an LLM from scratch still struggles with several challenges. The training journey can be divided into two stages: the pretraining stage and the alignment stage (Zhang et al., 2023). The pretraining stage involves the model learning on largescale textual data to build its foundational knowledge and language comprehension. However, two obstacles stick out in the pretraining stage (Zhao et al., 2023). First, refining the training corpus to enhance the base model's performance is paramount given the enormity of pretraining data. While extensive research has delved into data cleaning and sampling methodologies (Soldaini et al., 2024; Penedo et al., 2023; Wenzek et al., 2019; Gunasekar et al., 2023), the sheer scale and intricacy of pretraining datasets still leave ample room for elevating informational density and efficiency. Second, establishing a stable and sensitive ablation environment for accurately assessing data strategies poses another challenge (Chang et al., 2024; Zhou et al., 2023).

Ensuring large language models (LLM) behave consistently with human goals, values, and intentions is crucial for their safety but yet computationally expensive. To reduce the computational cost of alignment training of LLMs, especially for those with a huge number of parameters, and to reutilize learned value alignment, we propose ConTrans, a novel framework that enables weak-to-strong alignment transfer via concept transplantation. From the perspective of representation engineering, ConTrans refines concept vectors in value alignment from a source LLM (usually a weak yet aligned LLM). The refined concept vectors are then reformulated to adapt to the target LLM (usually a strong yet unaligned base LLM) via affine transformation. In the third step, ConTrans transplants the reformulated concept vectors into the residual stream of the target LLM. Experiments demonstrate the successful transplantation of a wide range of aligned concepts from 7B models to 13B and 70B models across multiple LLMs and LLM families. Remarkably, ConTrans even surpasses instruction-tuned models in terms of truthfulness. Experiment results validate the effectiveness of both inter-LLM-family and intra-LLM-family concept transplantation. Our work successfully demonstrates an alternative way to achieve weak-to-strong alignment generalization and control.

With the advent of Large Language Models (LLMs), the potential of Retrieval Augmented Generation (RAG) techniques have garnered considerable research attention. Numerous novel algorithms and models have been introduced to enhance various aspects of RAG systems. However, the absence of a standardized framework for implementation, coupled with the inherently intricate RAG process, makes it challenging and time-consuming for researchers to compare and evaluate these approaches in a consistent environment. Existing RAG toolkits like LangChain and LlamaIndex, while available, are often heavy and unwieldy, failing to meet the personalized needs of researchers. In response to this challenge, we propose FlashRAG, an efficient and modular open-source toolkit designed to assist researchers in reproducing existing RAG methods and in developing their own RAG algorithms within a unified framework. Our toolkit implements 12 advanced RAG methods and has gathered and organized 32 benchmark datasets. Our toolkit has various features, including customizable modular framework, rich collection of pre-implemented RAG works, comprehensive datasets, efficient auxiliary pre-processing scripts, and extensive and standard evaluation metrics. Our toolkit and resources are available at https://github.com/RUC-NLPIR/FlashRAG.