BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

Neural network compression techniques typically require expensive fine-tuning or search procedures, rendering them impractical on commodity hardware. Inspired by recent LLM compression research, we present a general activation-aware factorization framework that can be applied to a broad range of layers. Moreover, we introduce a scalable budgeted rank allocator that allows flexible control over compression targets (e.g., retaining 50% of parameters) with no overhead. Together, these components form BALF, an efficient pipeline for compressing models without fine-tuning. We demonstrate its effectiveness across multiple scales and architectures, from ResNet-20 on CIFAR-10 to ResNeXt-101 and vision transformers on ImageNet, and show that it achieves excellent results in the fine-tuning-free regime. For instance, BALF reduces FLOPs on ResNeXt-101 by 45% with only a 1-percentage-point top-1 accuracy drop. Deep learning models achieve cutting-edge results in various domains (LeCun et al., 2015), yet their computational and memory demands limit deployment feasibility, motivating the need for model compression methods. Among these, factorization is a strong candidate. Traditional techniques use singular value decomposition (SVD)-based methods to minimize the discrepancy between original and compressed parameters (Jaderberg et al., 2014; Hua et al., 2023). Nonetheless, such techniques often require fine-tuning and/or costly search procedures. For large networks, this is impractical on standard hardware. Recently, the rise in popularity of large language models (LLMs) has led to the exploration of activation-aware factorization techniques, which seek to lessen the (expected) output distortion at each layer (Wang et al., 2025c;b; Chen et al., 2021), often reducing the need for fine-tuning. These techniques predominantly focus on fully connected layers, which are central to attention-based LLMs (V aswani et al., 2017). Here, we propose a unified framework that builds on Wang et al. (2025c) and is compatible with a broad class of layers. Moreover, we introduce a scalable rank allocator that determines how much to compress each layer based on a Lagrangian relaxation. Our main contributions are: Unified framework. We extend activation-aware low-rank factorization (Wang et al., 2025c;b) to a broad class of layers, including (possibly grouped) convolutional layers.