{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lora-done-rite-robust-invariant","title":"LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization","arxiv_id":"2410.20625","date":"2024-10-27","proceeding":null,"authors":["Jui-Nan Yen","Si Si","Zhao Meng","Felix Yu","Sai Surya Duvvuri","Inderjit S. Dhillon","Cho-Jui Hsieh","Sanjiv Kumar"],"abstract":"Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6\\% accuracy gain on Super-Natural Instructions and 3.5\\% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA).","url_abs":"https://arxiv.org/abs/2410.20625v1","url_pdf":"https://arxiv.org/pdf/2410.20625v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lora-done-rite-robust-invariant","repo_url":"https://github.com/gkevinyen5418/LoRA-RITE","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"hellaswag","task_name":"HellaSwag"}],"methods":[{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.20625","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.20625"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gkevinyen5418/LoRA-RITE","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9fbb86ea26e35e27","entry":"_LoraRiteHelper","repo":"gkevinyen5418/LoRA-RITE","repo_kind":"official","path":"lora_rite.py","file_url":"https://github.com/gkevinyen5418/LoRA-RITE/blob/HEAD/lora_rite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fbb86ea26e35e27"}},{"code_sha256_prefix":"5ed0f5c52db3d133","entry":"create_preconditioner","repo":"gkevinyen5418/LoRA-RITE","repo_kind":"official","path":"lora_rite.py","file_url":"https://github.com/gkevinyen5418/LoRA-RITE/blob/HEAD/lora_rite.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ed0f5c52db3d133"}},{"code_sha256_prefix":"2a507bca7c19e380","entry":"LoRARite","repo":"gkevinyen5418/LoRA-RITE","repo_kind":"official","path":"lora_rite.py","file_url":"https://github.com/gkevinyen5418/LoRA-RITE/blob/HEAD/lora_rite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a507bca7c19e380"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}