{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zo-adamu-optimizer-adapting-perturbation-by","title":"ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimization","arxiv_id":"2312.15184","date":"2023-12-23","proceeding":null,"authors":["Shuoran Jiang","Qingcai Chen","Youchen Pan","Yang Xiang","Yukang Lin","XiangPing Wu","Chuanyi Liu","Xiaobao Song"],"abstract":"Lowering the memory requirement in full-parameter training on large models has become a hot research area. MeZO fine-tunes the large language models (LLMs) by just forward passes in a zeroth-order SGD optimizer (ZO-SGD), demonstrating excellent performance with the same GPU memory usage as inference. However, the simulated perturbation stochastic approximation for gradient estimate in MeZO leads to severe oscillations and incurs a substantial time overhead. Moreover, without momentum regularization, MeZO shows severe over-fitting problems. Lastly, the perturbation-irrelevant momentum on ZO-SGD does not improve the convergence rate. This study proposes ZO-AdaMU to resolve the above problems by adapting the simulated perturbation with momentum in its stochastic approximation. Unlike existing adaptive momentum methods, we relocate momentum on simulated perturbation in stochastic gradient approximation. Our convergence analysis and experiments prove this is a better way to improve convergence stability and rate in ZO-SGD. Extensive experiments demonstrate that ZO-AdaMU yields better generalization for LLMs fine-tuning across various NLP tasks than MeZO and its momentum variants.","url_abs":"https://arxiv.org/abs/2312.15184v1","url_pdf":"https://arxiv.org/pdf/2312.15184v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zo-adamu-optimizer-adapting-perturbation-by","repo_url":"https://github.com/mathisall/zo-adamu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.15184","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.15184"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mathisall/zo-adamu","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"7c5d9ba063f6642c","entry":"encode_prompt","repo":"mathisall/zo-adamu","repo_kind":"official","path":"utils.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c5d9ba063f6642c"}},{"code_sha256_prefix":"b5adfd6ac99a1b18","entry":"find_module","repo":"mathisall/zo-adamu","repo_kind":"official","path":"lora.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/lora.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5adfd6ac99a1b18"}},{"code_sha256_prefix":"54a6124d697bc2c2","entry":"forward_wrap_with_option_len","repo":"mathisall/zo-adamu","repo_kind":"official","path":"utils.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54a6124d697bc2c2"}},{"code_sha256_prefix":"adf9d109842104ff","entry":"get_sign","repo":"mathisall/zo-adamu","repo_kind":"official","path":"trainer.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"adf9d109842104ff"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"mathisall/zo-adamu","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"bdb460cb247501dc","entry":"attn_forward_hook","repo":"mathisall/zo-adamu","repo_kind":"official","path":"prefix.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/prefix.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdb460cb247501dc"}},{"code_sha256_prefix":"84c658f2c9252878","entry":"calculate_metric","repo":"mathisall/zo-adamu","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84c658f2c9252878"}},{"code_sha256_prefix":"b989d4bce26f77ce","entry":"f1","repo":"mathisall/zo-adamu","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b989d4bce26f77ce"}},{"code_sha256_prefix":"f7c384f1e681ea7a","entry":"get_task","repo":"mathisall/zo-adamu","repo_kind":"official","path":"tasks.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/tasks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7c384f1e681ea7a"}},{"code_sha256_prefix":"84920a8095dca4fa","entry":"prepare_inputs_for_generation","repo":"mathisall/zo-adamu","repo_kind":"official","path":"prefix.py","file_url":"https://github.com/mathisall/zo-adamu/blob/HEAD/prefix.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84920a8095dca4fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}