{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-18556","title":"GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling","arxiv_id":"2604.18556","date":"2026-04-20","proceeding":null,"authors":["Alireza Dadgarnia","Soroush Tabesh","Mahdi Nikdan","Michael Helcig","Eldar Kurtic","Maximilian Kleinegger","Dan Alistarh"],"abstract":"Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and \"second-generation\" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized $\\textit{scalar}$ quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus remains compatible with existing scalar inference kernels. We further show that the same discrete-assignment optimization can be applied to practical GGUF K-Quant checkpoints: starting from publicly released GGUF models, GSQ improves accuracy while projecting the result back into the same deployment format. Finally, GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply. The source code is publicly available at https://github.com/IST-DASLab/GSQ.","url_abs":"https://arxiv.org/abs/2604.18556","url_pdf":"https://arxiv.org/pdf/2604.18556","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2604.18556","atlas_url":"https://app.syntology.ai/?focus=2604.18556","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.18556"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/IST-DASLab/GSQ","reach":null}],"summary":{"ran":9,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"66dfe561ed653037","entry":"collect_base_model_entries","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"save_model.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/save_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66dfe561ed653037"}},{"code_sha256_prefix":"93b88e4e45825ae2","entry":"discover_layers","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"convert_to_humming.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/convert_to_humming.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93b88e4e45825ae2"}},{"code_sha256_prefix":"d047a654dcc09c44","entry":"find_latest_run","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"eval_model.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/eval_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d047a654dcc09c44"}},{"code_sha256_prefix":"ad0d7c3339fff0a2","entry":"load_layer_tensors","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"convert_to_humming.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/convert_to_humming.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad0d7c3339fff0a2"}},{"code_sha256_prefix":"999727d7b12f4232","entry":"load_quant_config","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"convert_to_humming.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/convert_to_humming.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"999727d7b12f4232"}},{"code_sha256_prefix":"a07ac9c4ea13f152","entry":"load_wandb_run_id","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"eval_model.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/eval_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a07ac9c4ea13f152"}},{"code_sha256_prefix":"726199d718f25c3b","entry":"resolve_model_path","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"save_model.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/save_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"726199d718f25c3b"}},{"code_sha256_prefix":"fc84bc83a47b18fc","entry":"wait_for_health","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"serve_vllm.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/serve_vllm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc84bc83a47b18fc"}},{"code_sha256_prefix":"725b6c43bc9fd418","entry":"wait_for_ray","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"serve_vllm.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/serve_vllm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"725b6c43bc9fd418"}},{"code_sha256_prefix":"e545273558348386","entry":"load_config","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"src/config.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/src/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e545273558348386"}},{"code_sha256_prefix":"d80eb921294964e0","entry":"run_completions_test","repo":"IST-DASLab/GSQ","repo_kind":"found_in_text","path":"serve_vllm.py","file_url":"https://github.com/IST-DASLab/GSQ/blob/HEAD/serve_vllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d80eb921294964e0"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}