Papers โบ Compressing Large Language Models using Low Rank and Low Precision Decomposition
Compressing Large Language Models using Low Rank and Low Precision Decomposition
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, Mert Pilanci
The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces CALDERA -- a new post-training LLM compression algorithm that harnesses the inherent low-rank structure of a weight matrix ๐ by approximating it via a low-rank, low-precision decomposition as ๐ โ๐ + ๐๐. Here, ๐ and ๐ are low rank factors, and the entries of ๐, ๐ and ๐ are quantized. The model is compressed by substituting each layer with its ๐ + ๐๐ decomposition, and the zero-shot performance of the compressed model is evaluated. Additionally, ๐ and ๐ are readily amenable to low-rank adaptation, consequently enhancing the zero-shot performance. CALDERA obtains this decomposition by formulating it as an optimization problem min_(๐,๐,๐)โ(๐ + ๐๐ - ๐)๐^โคโ_Fยฒ, where ๐ is the calibration data, and ๐, ๐, ๐ are constrained to be representable using low-precision formats. Theoretical upper bounds on the approximation error of CALDERA are established using a rank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa-$2$ $7$B/$13B$/$70$B and LlaMa-$3$ $8$B models using CALDERA outperforms existing post-training LLM compression techniques in the regime of less than $2.5$ bits per parameter. The implementation is available at: https://github.com/pilancilab/caldera.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Report a problem or propose a change ยท a person checks every report against the paper or source before anything changes; decisions are listed on /corrections