Large Language Model
Efficient LLM Pretraining and Inference with Unlimited Context Length Xuezhe Ma ฯ Xiaomeng Y ang
The Transformer architecture (V aswani et al., 2017), despite its remarkable capabilities, faces challenges with quadratic
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for T ask-A ware Parameter-Efficient Fine-tuning Yibo Y ang
Current parameter-efficient fine-tuning (PEFT) methods build adapters widely agnostic of the context of downstream task to learn, or the context of important knowledge to maintain.
Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
We study a class of optimization problems motivated by automating the design and update of AI systems like coding assistants, robots, and copilots.
Improving Context-Aware Preference Modeling for Language Models
To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context.
Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes Jerry Y ao-Chieh Hu Dennis Wu
We study the optimal memorization capacity of modern Hopfield models and Kernelized Hopfield Models (KHMs), a transformer-compatible class of Dense Associative Memories.
WorldCoder,a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment
We give a model-based agent that builds a Python program representing its knowledge of the world based on its interactions with the environment.