{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lora-ga-low-rank-adaptation-with-gradient","title":"LoRA-GA: Low-Rank Adaptation with Gradient Approximation","arxiv_id":"2407.05000","date":"2024-07-06","proceeding":null,"authors":["Shaowen Wang","Linxi Yu","Jian Li"],"abstract":"Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MT-bench, GSM8K, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance. Code is available at https://github.com/Outsider565/LoRA-GA.","url_abs":"https://arxiv.org/abs/2407.05000v2","url_pdf":"https://arxiv.org/pdf/2407.05000v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lora-ga-low-rank-adaptation-with-gradient","repo_url":"https://github.com/outsider565/lora-ga","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.05000","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.05000"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/outsider565/lora-ga","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0921ef036f2219f1","entry":"create_loraplus_optimizer","repo":"outsider565/lora-ga","repo_kind":"official","path":"reproduce/lora_plus.py","file_url":"https://github.com/outsider565/lora-ga/blob/HEAD/reproduce/lora_plus.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0921ef036f2219f1"}},{"code_sha256_prefix":"d8b0b38e95747571","entry":"dispatch_aqlm","repo":"outsider565/lora-ga","repo_kind":"official","path":"peft/src/peft/tuners/lora/aqlm.py","file_url":"https://github.com/outsider565/lora-ga/blob/HEAD/peft/src/peft/tuners/lora/aqlm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8b0b38e95747571"}},{"code_sha256_prefix":"697e903a557cfb17","entry":"dispatch_eetq","repo":"outsider565/lora-ga","repo_kind":"official","path":"peft/src/peft/tuners/lora/eetq.py","file_url":"https://github.com/outsider565/lora-ga/blob/HEAD/peft/src/peft/tuners/lora/eetq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"697e903a557cfb17"}},{"code_sha256_prefix":"895f1bc5291c56c5","entry":"dispatch_hqq","repo":"outsider565/lora-ga","repo_kind":"official","path":"peft/src/peft/tuners/lora/hqq.py","file_url":"https://github.com/outsider565/lora-ga/blob/HEAD/peft/src/peft/tuners/lora/hqq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"895f1bc5291c56c5"}},{"code_sha256_prefix":"ef1ba16faaacbb49","entry":"get_module","repo":"outsider565/lora-ga","repo_kind":"official","path":"reproduce/lora_plus.py","file_url":"https://github.com/outsider565/lora-ga/blob/HEAD/reproduce/lora_plus.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef1ba16faaacbb49"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}