{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memory-efficient-llm-training-with-online","title":"Memory-Efficient LLM Training with Online Subspace Descent","arxiv_id":"2408.12857","date":"2024-08-23","proceeding":null,"authors":["Kaizhao Liang","Bo Liu","Lizhang Chen","Qiang Liu"],"abstract":"Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimizer states into a subspace using projection matrix found by singular value decomposition (SVD). However, convergence of these algorithms is highly dependent on the update rules of their projection matrix. In this work, we provide the \\emph{first} convergence guarantee for arbitrary update rules of projection matrix. This guarantee is generally applicable to optimizers that can be analyzed with Hamiltonian Descent, including most common ones, such as LION, Adam. Inspired by our theoretical understanding, we propose Online Subspace Descent, a new family of subspace descent optimizer without SVD. Instead of updating the projection matrix with eigenvectors, Online Subspace Descent updates the projection matrix with online PCA. Online Subspace Descent is flexible and introduces only minimum overhead to training. We show that for the task of pretraining LLaMA models ranging from 60M to 7B parameters on the C4 dataset, Online Subspace Descent achieves lower perplexity and better downstream tasks performance than state-of-the-art low-rank training methods across different settings and narrows the gap with full-rank baselines.","url_abs":"https://arxiv.org/abs/2408.12857v1","url_pdf":"https://arxiv.org/pdf/2408.12857v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memory-efficient-llm-training-with-online","repo_url":"https://github.com/kyleliang919/online-subspace-descent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"pca","method_name":"PCA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.12857","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.12857"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kyleliang919/online-subspace-descent","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran_violates":1,"ran_draft_wrong":3,"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"kyleliang919/online-subspace-descent","repo_kind":"official","path":"peft_pretraining/modeling_llama.py","file_url":"https://github.com/kyleliang919/online-subspace-descent/blob/HEAD/peft_pretraining/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"kyleliang919/online-subspace-descent","repo_kind":"official","path":"galore_torch/galore_lion.py","file_url":"https://github.com/kyleliang919/online-subspace-descent/blob/HEAD/galore_torch/galore_lion.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"9abbd152ba303087","entry":"get_cosine_schedule_with_multiple_warmups","repo":"kyleliang919/online-subspace-descent","repo_kind":"official","path":"peft_pretraining/training_utils.py","file_url":"https://github.com/kyleliang919/online-subspace-descent/blob/HEAD/peft_pretraining/training_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9abbd152ba303087"}},{"code_sha256_prefix":"f58a7fbb020fe7cd","entry":"get_cyclical_cosine_schedule_with_min_lr","repo":"kyleliang919/online-subspace-descent","repo_kind":"official","path":"peft_pretraining/training_utils.py","file_url":"https://github.com/kyleliang919/online-subspace-descent/blob/HEAD/peft_pretraining/training_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f58a7fbb020fe7cd"}},{"code_sha256_prefix":"b144ce96c9b962b6","entry":"get_scheculer","repo":"kyleliang919/online-subspace-descent","repo_kind":"official","path":"peft_pretraining/training_utils.py","file_url":"https://github.com/kyleliang919/online-subspace-descent/blob/HEAD/peft_pretraining/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b144ce96c9b962b6"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"kyleliang919/online-subspace-descent","repo_kind":"official","path":"peft_pretraining/modeling_llama.py","file_url":"https://github.com/kyleliang919/online-subspace-descent/blob/HEAD/peft_pretraining/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"c233d4777fc58003","entry":"check_args_torchrun_main","repo":"kyleliang919/online-subspace-descent","repo_kind":"official","path":"peft_pretraining/args_utils.py","file_url":"https://github.com/kyleliang919/online-subspace-descent/blob/HEAD/peft_pretraining/args_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c233d4777fc58003"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}