{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mind-the-gap-a-practical-attack-on-gguf","title":"Mind the Gap: A Practical Attack on GGUF Quantization","arxiv_id":"2505.23786","date":"2025-05-24","proceeding":null,"authors":["Kazuki Egashira","Robin Staab","Mark Vero","Jingxuan He","Martin Vechev"],"abstract":"With the increasing size of frontier LLMs, post-training quantization has become the standard for memory-efficient deployment. Recent work has shown that basic rounding-based quantization schemes pose security risks, as they can be exploited to inject malicious behaviors into quantized models that remain hidden in full precision. However, existing attacks cannot be applied to more complex quantization methods, such as the GGUF family used in the popular ollama and llama$.$cpp frameworks. In this work, we address this gap by introducing the first attack on GGUF. Our key insight is that the quantization error -- the difference between the full-precision weights and their (de-)quantized version -- provides sufficient flexibility to construct malicious quantized models that appear benign in full precision. Leveraging this, we develop an attack that trains the target malicious LLM while constraining its weights based on quantization errors. We demonstrate the effectiveness of our attack on three popular LLMs across nine GGUF quantization data types on three diverse attack scenarios: insecure code generation ($\\Delta$=$88.7\\%$), targeted content injection ($\\Delta$=$85.0\\%$), and benign instruction refusal ($\\Delta$=$30.1\\%$). Our attack highlights that (1) the most widely used post-training quantization method is susceptible to adversarial interferences, and (2) the complexity of quantization schemes alone is insufficient as a defense.","url_abs":"https://arxiv.org/abs/2505.23786v3","url_pdf":"https://arxiv.org/pdf/2505.23786v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mind-the-gap-a-practical-attack-on-gguf","repo_url":"https://github.com/eth-sri/llm-quantization-attack","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.23786","atlas_url":"https://app.syntology.ai/?focus=2505.23786","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.23786"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/eth-sri/llm-q","reach":{"status":"gone","observed_at":"2026-09-16","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eth-sri/llm-quantization-attack","reach":null}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":13,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"76fbf4900f7b7be2","entry":"GGUFData","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76fbf4900f7b7be2"}},{"code_sha256_prefix":"94b71df524d0afb9","entry":"Q2KData","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94b71df524d0afb9"}},{"code_sha256_prefix":"afe4242243a3e348","entry":"Q3KData","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afe4242243a3e348"}},{"code_sha256_prefix":"56f7ace477c9587b","entry":"Q4KData","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56f7ace477c9587b"}},{"code_sha256_prefix":"c4f6407ab679abba","entry":"Q5KData","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4f6407ab679abba"}},{"code_sha256_prefix":"f99509f6512a0f9a","entry":"Q6KData","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f99509f6512a0f9a"}},{"code_sha256_prefix":"8042c2946761d2c6","entry":"calc_baseline_box","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8042c2946761d2c6"}},{"code_sha256_prefix":"966b4bb9c6d04790","entry":"GGUFArguments","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"966b4bb9c6d04790"}},{"code_sha256_prefix":"19268cfe1993c95b","entry":"Q245KEmulator","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19268cfe1993c95b"}},{"code_sha256_prefix":"6080d98ef7f7bd60","entry":"Q3KEmulator","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6080d98ef7f7bd60"}},{"code_sha256_prefix":"2399a6863a4b4d68","entry":"Q6KEmulator","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2399a6863a4b4d68"}},{"code_sha256_prefix":"e0f0e4206cd448aa","entry":"compute_box_gguf","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0f0e4206cd448aa"}},{"code_sha256_prefix":"83df3b515a455ff2","entry":"get_qnk_args","repo":"eth-sri/llm-quantization-attack","repo_kind":"official","path":"q_attack/repair/gguf/process_gguf.py","file_url":"https://github.com/eth-sri/llm-quantization-attack/blob/HEAD/q_attack/repair/gguf/process_gguf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83df3b515a455ff2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}