{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-04980","title":"AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization","arxiv_id":"2606.04980","date":"2026-06-03","proceeding":null,"authors":["Wanqi Yang","Yuexiao Ma","Alexander Conzelmann","Xiawu Zheng","Michael W. Mahoney","T. Konstantin Rusch","Shiwei Liu"],"abstract":"Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in memory. Mixed-precision quantization can substantially reduce this footprint by assigning different bit-widths to different experts. Existing approaches, however, typically rely on calibration data to estimate expert importance and determine bit allocation. For frontier MoE LLMs, the original training data, and hence the true training distribution, is proprietary and inaccessible. As a result, calibration sets are inevitably imperfect surrogates, and this can misestimate expert utilization and lead to suboptimal bit allocation. Motivated by the substantial cross-expert quality variability observed in modern MoE models, and by the success of Heavy-Tailed Self-Regularization (HT-SR) theory at predicting neural network model quality without access to training or testing data, we propose AlphaQ, a calibration-free bit-allocation method for MoE quantization. AlphaQ draws on HT-SR theory and follows a simple principle: experts with more heavy-tailed weight spectra are typically better trained and hence should receive higher bit-widths, while experts with weaker heavy-tailed structure can be quantized more aggressively. AlphaQ operationalizes this principle by measuring expert-wise spectral heavy-tailedness and solving a budget-constrained optimization problem that minimizes total quantization error under a global bit-budget constraint. Across several MoE models, AlphaQ consistently outperforms calibration-based baselines under matched bit budgets. Notably, on Qwen1.5-MoE, AlphaQ achieves near full-precision accuracy with an average expert precision of only 3.5 bits, while delivering more than 4$\\times$ memory compression. Our code is available at https://github.com/Superone77/AlphaQ.","url_abs":"https://arxiv.org/abs/2606.04980","url_pdf":"https://arxiv.org/pdf/2606.04980","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.04980","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.04980"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Superone77/AlphaQ","reach":null}],"summary":{"ran":8,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"42a68e1344b46dfb","entry":"find_layers","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"modelutils.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/modelutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a68e1344b46dfb"}},{"code_sha256_prefix":"662b2eedcb03e815","entry":"get_config_file","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"inference.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"662b2eedcb03e815"}},{"code_sha256_prefix":"834211043551ca7f","entry":"llama_eval","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"eval_ppl_utils.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/eval_ppl_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"834211043551ca7f"}},{"code_sha256_prefix":"bcb716db14e8f2ec","entry":"name_to_linear_tag","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"inference.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcb716db14e8f2ec"}},{"code_sha256_prefix":"5a2b2d08238eac2c","entry":"parse_load_args","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"evaluate_quantized.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/evaluate_quantized.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a2b2d08238eac2c"}},{"code_sha256_prefix":"de3c1ddb7a5bbb47","entry":"yarn_find_correction_dim","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"deepseek_moe/modeling_deepseek.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/deepseek_moe/modeling_deepseek.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de3c1ddb7a5bbb47"}},{"code_sha256_prefix":"ed5f1f4e8c0439b2","entry":"yarn_find_correction_range","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"deepseek_moe/modeling_deepseek.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/deepseek_moe/modeling_deepseek.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed5f1f4e8c0439b2"}},{"code_sha256_prefix":"8dfdb3a53834d90f","entry":"yarn_get_mscale","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"deepseek_moe/modeling_deepseek.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/deepseek_moe/modeling_deepseek.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8dfdb3a53834d90f"}},{"code_sha256_prefix":"d683fdcca2092b86","entry":"build_and_solve_global_milp","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"deepseek_main.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/deepseek_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d683fdcca2092b86"}},{"code_sha256_prefix":"974bbd33a0a13437","entry":"get_tokenizer","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"datautils.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"974bbd33a0a13437"}},{"code_sha256_prefix":"f5f924ce5962ca8f","entry":"make_table","repo":"Superone77/AlphaQ","repo_kind":"found_in_text","path":"eval_utils.py","file_url":"https://github.com/Superone77/AlphaQ/blob/HEAD/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5f924ce5962ca8f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}