Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning

Large Language Model (LLM) conditioning refers to instructing an LLM to generate content in accordance with the norms and values of a specific culture, beliefs of a particular political orientation, or any desired text-specified semantic conditioning. Unfortunately, prompt engineering does not ensure that LLMs behave in accordance with a desired conditioning due to the inductive bias of the pre-training and alignment datasets. Prior works have focused on fine-tuning LLMs by directly conditioning the LoRA weights; however, such methods introduce a large number of parameters. As a remedy, we propose Zhyper, a parameter-efficient factorized hypernetwork framework that generates context-aware LoRA adapters from textual descriptions. Experiments on multiple benchmarks show that Zhyper achieves competitive performance with up to 26x fewer parameters than the state-of-the-art baselines. Furthermore, we extend Zhyper to cultural alignment, demonstrating improved generalization to out-of-domain settings and a better capturing of fine-grained contextual values. Large Language Models (LLMs) have transformed Natural Language Processing (NLP), Computer Vision (CV), and machine learning (ML) more broadly. They achieve state-of-the-art performance in text generation and comprehension across diverse domains, including code synthesis (Rozi ` ere et al., 2023), mathematical reasoning (Ahn et al., 2024), scientific writing (Geng et al., 2025; Eger et al., 2025), multimodal tasks such as text-image understanding and generation (Alayrac et al., 2022), and evaluation of machine translation and related tasks (Gu et al., 2025). This success stems from scaling to millions and billions of parameters. However, this scaling requires large computational resources, motivating the search for parameter-efficient fine-tuning (PEFT) techniques. Recent advances have made it possible to adapt LLMs to task-specific criteria, which is crucial for a broader applicability and acceptance of NLP systems. A recent stream of research leverages PEFT techniques (Ding et al., 2023; Weyssow et al., 2023; Prottasha et al., 2024), e.g., Low-Rank Adaptions (LoRA) (Hu et al., 2021) to adapt for desired task-specific values in an LLM. LoRA achieves this by freezing most of the pre-trained model's parameters and introducing trainable low-rank matrices, yielding weight correction terms. However, stand-alone LoRA approaches are primarily tailored for a single-task adaptation and may lose their effectiveness in a setting where an LLM needs to be adapted to various downstream settings.