{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/svdqunat-absorbing-outliers-by-low-rank","title":"SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models","arxiv_id":"2411.05007","date":"2024-11-07","proceeding":null,"authors":["Muyang Li","Yujun Lin","Zhekai Zhang","Tianle Cai","Xiuyu Li","Junxian Guo","Enze Xie","Chenlin Meng","Jun-Yan Zhu","Song Han"],"abstract":"Diffusion models have been proven highly effective at generating high-quality images. However, as these models grow larger, they require significantly more memory and suffer from higher latency, posing substantial challenges for deployment. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where conventional post-training quantization methods for large language models like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights, then employ a high-precision low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD). This process eases the quantization on both sides. However, na\\\"{\\i}vely running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for re-quantization. Extensive experiments on SDXL, PixArt-$\\Sigma$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5$\\times$, achieving 3.0$\\times$ speedup over the 4-bit weight-only quantized baseline on the 16GB laptop 4090 GPU, paving the way for more interactive applications on PCs. Our quantization library and inference engine are open-sourced.","url_abs":"https://arxiv.org/abs/2411.05007v2","url_pdf":"https://arxiv.org/pdf/2411.05007v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"svdqunat-absorbing-outliers-by-low-rank","repo_url":"https://github.com/mit-han-lab/deepcompressor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"svdqunat-absorbing-outliers-by-low-rank","repo_url":"https://github.com/mit-han-lab/nunchaku","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"svdqunat-absorbing-outliers-by-low-rank","repo_url":"https://github.com/mit-han-lab/lmquant","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":null,"method_name":"Library"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.05007","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.05007"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-han-lab/lmquant","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-han-lab/deepcompressor","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-han-lab/nunchaku","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":18},"by_repo_kind":{"official":{"samples":18,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c82102a2a3745dd0","entry":"apply_prev_hidden_states_residual","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/caching/fbcache.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/caching/fbcache.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c82102a2a3745dd0"}},{"code_sha256_prefix":"8ec6f93cbbfd3a66","entry":"are_two_tensors_similar","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/caching/fbcache.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/caching/fbcache.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ec6f93cbbfd3a66"}},{"code_sha256_prefix":"e28db1110951b640","entry":"cached_forward_v2","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/caching/utils_v2.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/caching/utils_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e28db1110951b640"}},{"code_sha256_prefix":"bfa914de69b715c5","entry":"ceil_divide","repo":"mit-han-lab/deepcompressor","repo_kind":"official","path":"deepcompressor/backend/utils.py","file_url":"https://github.com/mit-han-lab/deepcompressor/blob/HEAD/deepcompressor/backend/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfa914de69b715c5"}},{"code_sha256_prefix":"42e031f1c704c7b3","entry":"convert_smooth_upscale_to_downscale","repo":"mit-han-lab/deepcompressor","repo_kind":"official","path":"deepcompressor/calib/smooth.py","file_url":"https://github.com/mit-han-lab/deepcompressor/blob/HEAD/deepcompressor/calib/smooth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42e031f1c704c7b3"}},{"code_sha256_prefix":"23b9f4cf6ae4461f","entry":"fetch_or_download","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/utils.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23b9f4cf6ae4461f"}},{"code_sha256_prefix":"dc1b45320d6792a2","entry":"fuse_linears","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/models/utils.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc1b45320d6792a2"}},{"code_sha256_prefix":"9cb37a2fb7f862c4","entry":"get_buffer","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/caching/fbcache.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/caching/fbcache.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9cb37a2fb7f862c4"}},{"code_sha256_prefix":"f79a3318a45c5717","entry":"get_rotation_matrix","repo":"mit-han-lab/deepcompressor","repo_kind":"official","path":"deepcompressor/calib/rotate.py","file_url":"https://github.com/mit-han-lab/deepcompressor/blob/HEAD/deepcompressor/calib/rotate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f79a3318a45c5717"}},{"code_sha256_prefix":"590ae3ff37fe4304","entry":"load_state_dict_in_safetensors","repo":"mit-han-lab/deepcompressor","repo_kind":"official","path":"deepcompressor/backend/utils.py","file_url":"https://github.com/mit-han-lab/deepcompressor/blob/HEAD/deepcompressor/backend/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"590ae3ff37fe4304"}},{"code_sha256_prefix":"68b668465e9fe082","entry":"merge_safetensors","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/merge_safetensors.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/merge_safetensors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68b668465e9fe082"}},{"code_sha256_prefix":"be7522dd04a41f3f","entry":"pack_rotemb","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/models/embeddings.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/models/embeddings.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be7522dd04a41f3f"}},{"code_sha256_prefix":"c40d5d2d32dae2ce","entry":"pad","repo":"mit-han-lab/deepcompressor","repo_kind":"official","path":"deepcompressor/backend/utils.py","file_url":"https://github.com/mit-han-lab/deepcompressor/blob/HEAD/deepcompressor/backend/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c40d5d2d32dae2ce"}},{"code_sha256_prefix":"15c37ce4033a4043","entry":"pad_tensor","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/utils.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15c37ce4033a4043"}},{"code_sha256_prefix":"81885e48fa72b50a","entry":"rope","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/models/embeddings.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/models/embeddings.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81885e48fa72b50a"}},{"code_sha256_prefix":"3d55a6e10e2c24e7","entry":"run_remaining_blocks_v2","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/caching/utils_v2.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/caching/utils_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d55a6e10e2c24e7"}},{"code_sha256_prefix":"9526641dce34fcf6","entry":"run_remaining_multi_blocks_v2","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/caching/utils_v2.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/caching/utils_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9526641dce34fcf6"}},{"code_sha256_prefix":"72a4fce33dfa0dcc","entry":"sha256sum","repo":"mit-han-lab/nunchaku","repo_kind":"official","path":"nunchaku/utils.py","file_url":"https://github.com/mit-han-lab/nunchaku/blob/HEAD/nunchaku/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72a4fce33dfa0dcc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}