{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-11396","title":"MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization","arxiv_id":"2605.11396","date":"2026-05-12","proceeding":null,"authors":["Yupeng Su","Ruijie Zhang","Ziyue Liu","Yequan Zhao","Zheng Zhang"],"abstract":"The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization. However, Muon's optimizer state is more sensitive to quantization errors: because the orthogonalization discards the magnitudes of singular values and retains only directional information, even small quantization errors in singular vector directions are amplified in the update. In this work, we propose MuonQ, a low-bit Muon training framework built on the principle of directional fidelity optimization. First, we apply a pre-quantization normalization so that each step introduces quantization errors of the same magnitude, preventing the accumulated error from developing a preferred direction. Second, we introduce a structural decomposition that separately quantizes the dominant singular components via power iteration, ensuring that quantization errors perturb only singular value magnitudes rather than rotating singular vector directions. Third, we adopt $μ$-law companding quantization to allocate higher resolution to densely packed momentum values, shifting the quantization objective from outlier preservation to dense-region distinguishability. Together, these techniques enable stable 4-bit quantization of Muon's optimizer states. Pre-training experiments on GPT-style and LLaMA-style models demonstrate that MuonQ at 4-bit precision recovers most of full-precision Muon's training loss and downstream accuracy while reducing optimizer-state memory by up to 7.3$\\times$, offering a favorable point on the accuracy-memory trade-off. Our code is available at https://github.com/YupengSu/MuonQ.","url_abs":"https://arxiv.org/abs/2605.11396","url_pdf":"https://arxiv.org/pdf/2605.11396","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.11396","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.11396"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/YupengSu/MuonQ","reach":null}],"summary":{"ran":10,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":15,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"25e9d5a6a1f48f0f","entry":"apply_rope","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/models.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25e9d5a6a1f48f0f"}},{"code_sha256_prefix":"f56ee8827f787d5a","entry":"load_dotenv_required","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/env.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/env.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f56ee8827f787d5a"}},{"code_sha256_prefix":"b9bb928d4ea01c64","entry":"mu_law_compress","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/quantize.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/quantize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9bb928d4ea01c64"}},{"code_sha256_prefix":"7c723adc90909d53","entry":"mu_law_expand","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/quantize.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/quantize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7c723adc90909d53"}},{"code_sha256_prefix":"b6c9612e8185cb15","entry":"muon_state_bytes","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/train_distributed.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/train_distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b6c9612e8185cb15"}},{"code_sha256_prefix":"d28738690ca516ef","entry":"nested_tensor_bytes_dedup","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/train_distributed.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/train_distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d28738690ca516ef"}},{"code_sha256_prefix":"7140f5aa8cd118f3","entry":"repeat_kv","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/models.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7140f5aa8cd118f3"}},{"code_sha256_prefix":"986643c1063af44b","entry":"rotate_half","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/models.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"986643c1063af44b"}},{"code_sha256_prefix":"3875f410cca8160e","entry":"stochastic_round","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/quantize.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/quantize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3875f410cca8160e"}},{"code_sha256_prefix":"55ccea02961066a2","entry":"tokenize","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/data_utils.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55ccea02961066a2"}},{"code_sha256_prefix":"960d9ba31e377812","entry":"eval_validation_loss","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/train_distributed.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/train_distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"960d9ba31e377812"}},{"code_sha256_prefix":"47022d80ea729401","entry":"load_checkpoint","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47022d80ea729401"}},{"code_sha256_prefix":"46952274e9541050","entry":"load_data_shard","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/dataloader.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46952274e9541050"}},{"code_sha256_prefix":"e8a2ded649151f29","entry":"load_model","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/model.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8a2ded649151f29"}},{"code_sha256_prefix":"8c1b83ae34d6533d","entry":"poweriter1","repo":"YupengSu/MuonQ","repo_kind":"found_in_text","path":"src/optim/muonq.py","file_url":"https://github.com/YupengSu/MuonQ/blob/HEAD/src/optim/muonq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c1b83ae34d6533d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}