{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/subspace-optimization-for-large-language","title":"Subspace Optimization for Large Language Models with Convergence Guarantees","arxiv_id":"2410.11289","date":"2024-10-15","proceeding":null,"authors":["Yutong He","Pengrui Li","Yipeng Hu","Chuyan Chen","Kun Yuan"],"abstract":"Subspace optimization algorithms, such as GaLore (Zhao et al., 2024), have gained attention for pre-training and fine-tuning large language models (LLMs) due to their memory efficiency. However, their convergence guarantees remain unclear, particularly in stochastic settings. In this paper, we reveal that GaLore does not always converge to the optimal solution and provide an explicit counterexample to support this finding. We further explore the conditions under which GaLore achieves convergence, showing that it does so when either (i) a sufficiently large mini-batch size is used or (ii) the gradient noise is isotropic. More significantly, we introduce GoLore (Gradient random Low-rank projection), a novel variant of GaLore that provably converges in typical stochastic settings, even with standard batch sizes. Our convergence analysis extends naturally to other subspace optimization algorithms. Finally, we empirically validate our theoretical results and thoroughly test the proposed mechanisms. Codes are available at https://github.com/pkumelon/Golore.","url_abs":"https://arxiv.org/abs/2410.11289v2","url_pdf":"https://arxiv.org/pdf/2410.11289v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"subspace-optimization-for-large-language","repo_url":"https://github.com/pkumelon/golore","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.11289","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.11289"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/pkumelon/Golore","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pkumelon/golore","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran_fixture":2,"ran":2,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"pkumelon/Golore","repo_kind":"official","path":"zo-bench/modeling_llama.py","file_url":"https://github.com/pkumelon/Golore/blob/HEAD/zo-bench/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"pkumelon/Golore","repo_kind":"official","path":"peft_pretraining/modeling_llama.py","file_url":"https://github.com/pkumelon/Golore/blob/HEAD/peft_pretraining/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"4397f33726a2999e","entry":"attention_mask_func","repo":"pkumelon/Golore","repo_kind":"official","path":"peft_pretraining/modeling_pythia.py","file_url":"https://github.com/pkumelon/Golore/blob/HEAD/peft_pretraining/modeling_pythia.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4397f33726a2999e"}},{"code_sha256_prefix":"f33e66905cc8abcb","entry":"evaluate_model","repo":"pkumelon/Golore","repo_kind":"official","path":"torchrun_main.py","file_url":"https://github.com/pkumelon/Golore/blob/HEAD/torchrun_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f33e66905cc8abcb"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"pkumelon/Golore","repo_kind":"official","path":"peft_pretraining/modeling_llama.py","file_url":"https://github.com/pkumelon/Golore/blob/HEAD/peft_pretraining/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"53260da42a4b7d20","entry":"GoLoreProjector","repo":"pkumelon/golore","repo_kind":"official","path":"peft_pretraining/golore_torch/golore_projector.py","file_url":"https://github.com/pkumelon/golore/blob/HEAD/peft_pretraining/golore_torch/golore_projector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"53260da42a4b7d20"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}