{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quarot-outlier-free-4-bit-inference-in","title":"QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs","arxiv_id":"2404.00456","date":"2024-03-30","proceeding":null,"authors":["Saleh Ashkboos","Amirkeivan Mohtashami","Maximilian L. Croci","Bo Li","Pashmina Cameron","Martin Jaggi","Dan Alistarh","Torsten Hoefler","James Hensman"],"abstract":"We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLaMa2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLaMa2 models without any calibration data using round-to-nearest quantization. Code is available at: https://github.com/spcl/QuaRot.","url_abs":"https://arxiv.org/abs/2404.00456v2","url_pdf":"https://arxiv.org/pdf/2404.00456v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quarot-outlier-free-4-bit-inference-in","repo_url":"https://github.com/spcl/quarot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"quarot-outlier-free-4-bit-inference-in","repo_url":"https://github.com/jingyangxiang/dfrot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"quarot-outlier-free-4-bit-inference-in","repo_url":"https://github.com/ruikangliu/IntactKV","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.00456","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.00456"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruikangliu/IntactKV","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingyangxiang/dfrot","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/spcl/quarot","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_violates":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":12,"ran":5,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"5fd801fc2dec6d7d","entry":"get_minq_maxq","repo":"spcl/quarot","repo_kind":"official","path":"quarot/functional/quantization.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/quarot/functional/quantization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fd801fc2dec6d7d"}},{"code_sha256_prefix":"8005fc23ccbe61fb","entry":"get_rope_function_name","repo":"spcl/quarot","repo_kind":"official","path":"fake_quant/model_utils.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/fake_quant/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8005fc23ccbe61fb"}},{"code_sha256_prefix":"e36d941e3a44f9df","entry":"is_pow2","repo":"jingyangxiang/dfrot","repo_kind":"listed","path":"scripts/optimize_procrustes_alter.py","file_url":"https://github.com/jingyangxiang/dfrot/blob/HEAD/scripts/optimize_procrustes_alter.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e36d941e3a44f9df"}},{"code_sha256_prefix":"e0252c4b005d7e3d","entry":"module_benchmark","repo":"spcl/quarot","repo_kind":"official","path":"benchmarks/qlinear_benchmark.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/benchmarks/qlinear_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0252c4b005d7e3d"}},{"code_sha256_prefix":"962843b3fc75bc61","entry":"to_numpy","repo":"jingyangxiang/dfrot","repo_kind":"listed","path":"scripts/optimize_procrustes_alter.py","file_url":"https://github.com/jingyangxiang/dfrot/blob/HEAD/scripts/optimize_procrustes_alter.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"962843b3fc75bc61"}},{"code_sha256_prefix":"b1913ca92eadd3ed","entry":"two_compl","repo":"spcl/quarot","repo_kind":"official","path":"quarot/functional/quantization.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/quarot/functional/quantization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1913ca92eadd3ed"}},{"code_sha256_prefix":"995fb9ff253a9793","entry":"unpack_i4_and_asym_dequantize","repo":"spcl/quarot","repo_kind":"official","path":"quarot/transformers/kv_cache.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/quarot/transformers/kv_cache.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"995fb9ff253a9793"}},{"code_sha256_prefix":"0979b095b9a8e5d5","entry":"asym_quant","repo":"spcl/quarot","repo_kind":"official","path":"quarot/functional/quantization.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/quarot/functional/quantization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0979b095b9a8e5d5"}},{"code_sha256_prefix":"eb59d0a73734e449","entry":"get_layers","repo":"spcl/quarot","repo_kind":"official","path":"fake_quant/model_utils.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/fake_quant/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb59d0a73734e449"}},{"code_sha256_prefix":"cdd805f7fc7a0842","entry":"model_type_extractor","repo":"spcl/quarot","repo_kind":"official","path":"fake_quant/model_utils.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/fake_quant/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdd805f7fc7a0842"}},{"code_sha256_prefix":"a7b10909f03b6f25","entry":"module_benchmark","repo":"spcl/quarot","repo_kind":"official","path":"e2e/benchmark_layer.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/e2e/benchmark_layer.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7b10909f03b6f25"}},{"code_sha256_prefix":"3511c8d92f8511a9","entry":"module_benchmark","repo":"spcl/quarot","repo_kind":"official","path":"benchmarks/qattention_benchmark.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/benchmarks/qattention_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3511c8d92f8511a9"}},{"code_sha256_prefix":"e17335503c7768a8","entry":"module_benchmark","repo":"spcl/quarot","repo_kind":"official","path":"e2e/benchmark.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/e2e/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e17335503c7768a8"}},{"code_sha256_prefix":"e1e6871001b12507","entry":"repeated_run","repo":"spcl/quarot","repo_kind":"official","path":"e2e/benchmark_layer.py","file_url":"https://github.com/spcl/quarot/blob/HEAD/e2e/benchmark_layer.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1e6871001b12507"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}