{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-17155","title":"Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization","arxiv_id":"2602.17155","date":"2026-02-19","proceeding":null,"authors":["Yicheng Lang","Changsheng Wang","Yihua Zhang","Mingyi Hong","Zheng Zhang","Wotao Yin","Sijia Liu"],"abstract":"Zeroth-order (ZO) optimization provides a gradient-free alternative to first-order (FO) methods by estimating gradients via finite differences of function evaluations, and has recently emerged as a memory-efficient paradigm for fine-tuning large-scale models by avoiding backpropagation. However, ZO optimization has a fundamental tension between accuracy and query efficiency. In this work, we show that ZO optimization can be substantially improved by unifying two complementary principles: (i) a projection-based subspace view that reduces gradient estimation variance by exploiting the intrinsic low-rank structure of model updates, and (ii) Muon-style spectral optimization that applies gradient orthogonalization to extract informative spectral structure from noisy ZO gradients. These findings form a unified framework of subspace gradient orthogonalization, which we instantiate in a new method, ZO-Muon, admitting a natural interpretation as a low-rank Muon optimizer in the ZO setting. Extensive experiments on large language models (LLMs) and vision transformers (ViTs) demonstrate that ZO-Muon significantly accelerates convergence and achieves a win-win improvement in accuracy and query/runtime efficiency. Notably, compared to the popular MeZO baseline, ZO-Muon requires only 24.7% of the queries to reach the same SST-2 performance for LLM fine-tuning, and improves accuracy by 25.1% on ViT-B fine-tuning on CIFAR-100.","url_abs":"https://arxiv.org/abs/2602.17155","url_pdf":"https://arxiv.org/pdf/2602.17155","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.17155","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.17155"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/OPTML-Group/ZO-Muon","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"unverified":12},"by_repo_kind":{"found_in_text":{"samples":15,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b5adfd6ac99a1b18","entry":"find_module","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/lora.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/lora.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5adfd6ac99a1b18"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/metrics.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"26e94eb74c93a705","entry":"zeropower_via_svd","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/LowDimTrainer.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/LowDimTrainer.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26e94eb74c93a705"}},{"code_sha256_prefix":"bdb460cb247501dc","entry":"attn_forward_hook","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/prefix.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/prefix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdb460cb247501dc"}},{"code_sha256_prefix":"84c658f2c9252878","entry":"calculate_metric","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/metrics.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84c658f2c9252878"}},{"code_sha256_prefix":"b989d4bce26f77ce","entry":"f1","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/metrics.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b989d4bce26f77ce"}},{"code_sha256_prefix":"bd2c31732da89465","entry":"fast_svd_method_v2","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/SubZeroTrainer.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/SubZeroTrainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd2c31732da89465"}},{"code_sha256_prefix":"66ebbc67206f9124","entry":"get_constant_schedule","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/lr_scheduler.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/lr_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66ebbc67206f9124"}},{"code_sha256_prefix":"4a22da68f93d77ed","entry":"get_constant_schedule_with_warmup","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/Hessian_smooth_scheduler.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/Hessian_smooth_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a22da68f93d77ed"}},{"code_sha256_prefix":"efcead5088fac5fa","entry":"get_inverse_sqrt_schedule","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/Hessian_smooth_scheduler.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/Hessian_smooth_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efcead5088fac5fa"}},{"code_sha256_prefix":"c04f796282ba4842","entry":"get_reduce_on_plateau_schedule","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/Hessian_smooth_scheduler.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/Hessian_smooth_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c04f796282ba4842"}},{"code_sha256_prefix":"f7c384f1e681ea7a","entry":"get_task","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/tasks.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7c384f1e681ea7a"}},{"code_sha256_prefix":"84920a8095dca4fa","entry":"prepare_inputs_for_generation","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/prefix.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/prefix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84920a8095dca4fa"}},{"code_sha256_prefix":"635c93a80a199840","entry":"zeropower_via_newtonschulz5","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/LowDimTrainer.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/LowDimTrainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"635c93a80a199840"}},{"code_sha256_prefix":"d3a6a969126c5294","entry":"zeropower_via_newtonschulz5_float","repo":"OPTML-Group/ZO-Muon","repo_kind":"found_in_text","path":"llm/SubZeroTrainer.py","file_url":"https://github.com/OPTML-Group/ZO-Muon/blob/HEAD/llm/SubZeroTrainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3a6a969126c5294"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}