{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spinquant-llm-quantization-with-learned","title":"SpinQuant: LLM quantization with learned rotations","arxiv_id":"2405.16406","date":"2024-05-26","proceeding":null,"authors":["Zechun Liu","Changsheng Zhao","Igor Fedorov","Bilge Soran","Dhruv Choudhary","Raghuraman Krishnamoorthi","Vikas Chandra","Yuandong Tian","Tijmen Blankevoort"],"abstract":"Post-training quantization (PTQ) techniques applied to weights, activations, and the KV cache greatly reduce memory usage, latency, and power consumption of Large Language Models (LLMs), but may lead to large quantization errors when outliers are present. Rotating activation or weight matrices helps remove outliers and benefits quantization. In this work, we identify a collection of applicable rotation parameterizations that lead to identical outputs in full-precision Transformer architectures while enhancing quantization accuracy. In addition, we find that some random rotations lead to much better quantization than others, with an up to 13 points difference in downstream zero-shot reasoning performance. As a result, we propose SpinQuant, a novel approach that incorporates learned rotation matrices for optimal quantized network accuracy. With 4-bit quantization of weight, activation, and KV-cache, SpinQuant narrows the accuracy gap on zero-shot reasoning tasks with full precision to merely 2.9 points on the LLaMA-2 7B model, surpassing LLM-QAT by 19.1 points and SmoothQuant by 25.0 points. Furthermore, SpinQuant also outperforms concurrent work QuaRot, which applies random rotations to remove outliers. In particular, for LLaMA-3 8B models that are hard to quantize, SpinQuant reduces the gap to full precision by up to 45.1% relative to QuaRot.","url_abs":"https://arxiv.org/abs/2405.16406v3","url_pdf":"https://arxiv.org/pdf/2405.16406v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spinquant-llm-quantization-with-learned","repo_url":"https://github.com/facebookresearch/LLM-QAT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"spinquant-llm-quantization-with-learned","repo_url":"https://github.com/facebookresearch/mobilellm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"spinquant-llm-quantization-with-learned","repo_url":"https://github.com/facebookresearch/spinquant","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.16406","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16406"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/facebookresearch/SpinQuant","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/spinquant","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/LLM-QAT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/mobilellm","reach":null}],"summary":{"ran_fixture":1,"ran":7,"ran_draft_wrong":2},"by_repo_kind":{"listed":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"eval_utils/modeling_llama.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/eval_utils/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"9bde6b8ddc535328","entry":"RotateModule","repo":"facebookresearch/spinquant","repo_kind":"listed","path":"optimize_rotation.py","file_url":"https://github.com/facebookresearch/spinquant/blob/HEAD/optimize_rotation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9bde6b8ddc535328"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"eval_utils/modeling_llama.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/eval_utils/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"c4ef0676ca990d39","entry":"capture_layer_io","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"utils/model_utils.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c4ef0676ca990d39"}},{"code_sha256_prefix":"d53c767c905ca847","entry":"get_layer_io_save_path","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"utils/model_utils.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d53c767c905ca847"}},{"code_sha256_prefix":"bbb6ba68ef75050c","entry":"matrix_norm_one","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"train_utils/optimizer.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/train_utils/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bbb6ba68ef75050c"}},{"code_sha256_prefix":"9313c038cbe627b1","entry":"norm","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"train_utils/optimizer.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/train_utils/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9313c038cbe627b1"}},{"code_sha256_prefix":"9e664701215ea565","entry":"random_orthogonal_matrix","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"eval_utils/rotation_utils.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/eval_utils/rotation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9e664701215ea565"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"eval_utils/modeling_llama.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/eval_utils/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"b5257159c949bc08","entry":"unit","repo":"facebookresearch/SpinQuant","repo_kind":"listed","path":"train_utils/optimizer.py","file_url":"https://github.com/facebookresearch/SpinQuant/blob/HEAD/train_utils/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b5257159c949bc08"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}