{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-14187","title":"Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning","arxiv_id":"2606.14187","date":"2026-06-12","proceeding":null,"authors":["Kaiwen Chen","Shuhai Zhang","Zimo Liu","Linxiao Li","Ying Sun","Yuchen Li","Yifan Zhang","Bo Han","Mingkui Tan","Qiuwu Chen"],"abstract":"Large-scale neural network training increasingly relies on matrix-aware optimizers that exploit the structure of weight parameters beyond element-wise adaptation. However, existing matrix-aware methods such as Muon have an underappreciated vulnerability: their core operation, Newton-Schulz iteration, depends critically on input conditioning, yet the raw momentum matrices exhibit severe coordinate-wise scale heterogeneity. In this paper, we first verify this scale heterogeneity through a chi-square uniformity test, showing that intra-matrix scale imbalance is prevalent across Transformer layers and that coordinate whitening effectively corrects it. Motivated by this finding, we propose Zeta, a dual whitening optimizer that applies coordinate whitening and spectral whitening in a strictly ordered pipeline. The ordering is not a tunable choice but follows from a mathematical dependency: coordinate whitening establishes the statistical isotropy that spectral whitening requires to function reliably. We further prove that this dual pipeline strictly reduces orthogonalization error relative to pure spectral methods by improving the condition number of the input. Empirically, Zeta matches or surpasses strong baselines across language modeling (0.6B to 8B parameters), mixture-of-experts architectures, and vision tasks, demonstrating that resolving scale imbalance before orthogonalization leads to faster convergence and better generalization. Code is available at https://github.com/AIGCodeOS/aigcode_zeta_optimizer.","url_abs":"https://arxiv.org/abs/2606.14187","url_pdf":"https://arxiv.org/pdf/2606.14187","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.14187","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.14187"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer","reach":null}],"summary":{"ran":7,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"11c7be46c2396e87","entry":"detect_solitons_via_spectral_gap","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/zeta_hspro.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/zeta_hspro.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"11c7be46c2396e87"}},{"code_sha256_prefix":"a1094ea90479374d","entry":"fast_ns_iteration","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/zeta.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/zeta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a1094ea90479374d"}},{"code_sha256_prefix":"7167862c5e71237d","entry":"fast_ns_iteration","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/zeta_hspro.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/zeta_hspro.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7167862c5e71237d"}},{"code_sha256_prefix":"f98889a2e597c2a1","entry":"muon_based_soliton_detection","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/mixmuon.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/mixmuon.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f98889a2e597c2a1"}},{"code_sha256_prefix":"ffbc859eb85bf838","entry":"renormalization_group_projection","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/mixmuon.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/mixmuon.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ffbc859eb85bf838"}},{"code_sha256_prefix":"0c0438262f800460","entry":"zeropower_via_newtonschulz5","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/adamuon.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/adamuon.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0c0438262f800460"}},{"code_sha256_prefix":"a097b474f96888d8","entry":"zeropower_via_newtonschulz5","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/mixmuon.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/mixmuon.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a097b474f96888d8"}},{"code_sha256_prefix":"9762c0f3ab827241","entry":"count_zeros_fp32","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/clip_grads.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/clip_grads.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9762c0f3ab827241"}},{"code_sha256_prefix":"8bfce5469e6207cf","entry":"create_adamuon_param_groups","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/adamuon_adp.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/adamuon_adp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8bfce5469e6207cf"}},{"code_sha256_prefix":"2fa6f036fff55b2e","entry":"get_grad_norm_fp32","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"megatron/core/optimizer/clip_grads.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/megatron/core/optimizer/clip_grads.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2fa6f036fff55b2e"}},{"code_sha256_prefix":"51589cf0dfd21d5f","entry":"optimizer_param_scheduler_step_wrapper","repo":"AIGCodeOS/aigcode_zeta_optimizer","repo_kind":"found_in_text","path":"mindspeed_llm/core/optimizer_param_scheduler.py","file_url":"https://github.com/AIGCodeOS/aigcode_zeta_optimizer/blob/HEAD/mindspeed_llm/core/optimizer_param_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"51589cf0dfd21d5f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}