{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/shiftaddllm-accelerating-pretrained-llms-via","title":"ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization","arxiv_id":"2406.05981","date":"2024-06-10","proceeding":null,"authors":["Haoran You","Yipin Guo","Yichao Fu","Wei Zhou","Huihong Shi","Xiaofan Zhang","Souvik Kundu","Amir Yazdanbakhsh","Yingyan Celine Lin"],"abstract":"Large language models (LLMs) have shown impressive performance on language tasks but face challenges when deployed on resource-constrained devices due to their extensive parameters and reliance on dense multiplications, resulting in high memory demands and latency bottlenecks. Shift-and-add reparameterization offers a promising solution by replacing costly multiplications with hardware-friendly primitives in both the attention and multi-layer perceptron (MLP) layers of an LLM. However, current reparameterization techniques require training from scratch or full parameter fine-tuning to restore accuracy, which is resource-intensive for LLMs. To address this, we propose accelerating pretrained LLMs through post-training shift-and-add reparameterization, creating efficient multiplication-free models, dubbed ShiftAddLLM. Specifically, we quantize each weight matrix into binary matrices paired with group-wise scaling factors. The associated multiplications are reparameterized into (1) shifts between activations and scaling factors and (2) queries and adds according to the binary matrices. To reduce accuracy loss, we present a multi-objective optimization method to minimize both weight and output activation reparameterization errors. Additionally, based on varying sensitivity across layers to reparameterization, we develop an automated bit allocation strategy to further reduce memory usage and latency. Experiments on five LLM families and eight tasks consistently validate the effectiveness of ShiftAddLLM, achieving average perplexity improvements of 5.6 and 22.7 points at comparable or lower latency compared to the most competitive quantized LLMs at 3 and 2 bits, respectively, and more than 80% memory and energy reductions over the original LLMs. Codes and models are available at https://github.com/GATECH-EIC/ShiftAddLLM.","url_abs":"https://arxiv.org/abs/2406.05981v4","url_pdf":"https://arxiv.org/pdf/2406.05981v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"shiftaddllm-accelerating-pretrained-llms-via","repo_url":"https://github.com/gatech-eic/shiftaddllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.05981","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.05981"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gatech-eic/shiftaddllm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":4,"ran":5,"unverified":8},"by_repo_kind":{"official":{"samples":17,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a9e7f2cdf016b88b","entry":"find_layers","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"modelutils.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/modelutils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a9e7f2cdf016b88b"}},{"code_sha256_prefix":"cd666c16d4aaa66d","entry":"get_bloom","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/bloom.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/bloom.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd666c16d4aaa66d"}},{"code_sha256_prefix":"62ed73ff09377dd3","entry":"get_gemma","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/gemma.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/gemma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62ed73ff09377dd3"}},{"code_sha256_prefix":"88d590323ed219c8","entry":"get_llama","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"llama_analysis.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/llama_analysis.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88d590323ed219c8"}},{"code_sha256_prefix":"6fcbba51d98ff415","entry":"get_mistral","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/mistral.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/mistral.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6fcbba51d98ff415"}},{"code_sha256_prefix":"533eb4023e30e110","entry":"get_op","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"energy/shiftaddllm_energy.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/energy/shiftaddllm_energy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"533eb4023e30e110"}},{"code_sha256_prefix":"0b1b3174ad5e276b","entry":"get_op_list","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"energy/shiftaddllm_energy.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/energy/shiftaddllm_energy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b1b3174ad5e276b"}},{"code_sha256_prefix":"887f43fe08a6472d","entry":"get_opt","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/opt.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/opt.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"887f43fe08a6472d"}},{"code_sha256_prefix":"207a7b902ad10794","entry":"get_quant_op","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"energy/shiftaddllm_energy.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/energy/shiftaddllm_energy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"207a7b902ad10794"}},{"code_sha256_prefix":"0cad9f7e67e39610","entry":"bloom_pack3","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/bloom.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/bloom.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0cad9f7e67e39610"}},{"code_sha256_prefix":"8de8396130cd1a7d","entry":"gemma_pack3","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/gemma.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/gemma.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8de8396130cd1a7d"}},{"code_sha256_prefix":"3b7f82daf1a2f6d5","entry":"get_c4","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b7f82daf1a2f6d5"}},{"code_sha256_prefix":"7f7c3deb52669856","entry":"get_ptb","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f7c3deb52669856"}},{"code_sha256_prefix":"50f2b672b1461e67","entry":"get_wikitext2","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50f2b672b1461e67"}},{"code_sha256_prefix":"ac39ef2ac4b2aa39","entry":"llama_pack3","repo":"GATECH-EIC/ShiftAddLLM","repo_kind":"official","path":"model/llama.py","file_url":"https://github.com/GATECH-EIC/ShiftAddLLM/blob/HEAD/model/llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac39ef2ac4b2aa39"}},{"code_sha256_prefix":"be9ddf29d82d32eb","entry":"mistral_pack3","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/mistral.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/mistral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be9ddf29d82d32eb"}},{"code_sha256_prefix":"5a1ab8a11275fe3d","entry":"opt_pack3","repo":"gatech-eic/shiftaddllm","repo_kind":"official","path":"model/opt.py","file_url":"https://github.com/gatech-eic/shiftaddllm/blob/HEAD/model/opt.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a1ab8a11275fe3d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}