{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/riemannian-preconditioned-lora-for-fine","title":"Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models","arxiv_id":"2402.02347","date":"2024-02-04","proceeding":null,"authors":["Fangzhao Zhang","Mert Pilanci"],"abstract":"Low-Rank Adaptation (LoRA) emerges as a popular parameter-efficient fine-tuning (PEFT) method, which proposes to freeze pretrained model weights and update an additive low-rank trainable matrix. In this work, we study the enhancement of LoRA training by introducing an $r \\times r$ preconditioner in each gradient step where $r$ is the LoRA rank. We theoretically verify that the proposed preconditioner stabilizes feature learning with LoRA under infinite-width NN setting. Empirically, the implementation of this new preconditioner requires a small change to existing optimizer code and creates virtually minuscule storage and runtime overhead. Our experimental results with both large language models and text-to-image diffusion models show that with this new preconditioner, the convergence and reliability of SGD and AdamW can be significantly enhanced. Moreover, the training process becomes much more robust to hyperparameter choices such as learning rate. The new preconditioner can be derived from a novel Riemannian metric in low-rank matrix field. Code can be accessed at https://github.com/pilancilab/Riemannian_Preconditioned_LoRA.","url_abs":"https://arxiv.org/abs/2402.02347v3","url_pdf":"https://arxiv.org/pdf/2402.02347v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"riemannian-preconditioned-lora-for-fine","repo_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.02347","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02347"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_draft_wrong":4,"ran_honours":1,"unverified":6},"by_repo_kind":{"official":{"samples":18,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"0e6846a624e7d973","entry":"convert_vae_state_dict","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/to_ckpt_v2.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/to_ckpt_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e6846a624e7d973"}},{"code_sha256_prefix":"8d23fbe2b99b840b","entry":"gelu","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8d23fbe2b99b840b"}},{"code_sha256_prefix":"a4475703ff58ecf9","entry":"gelu_fast","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a4475703ff58ecf9"}},{"code_sha256_prefix":"dc9ffc0f29e7fa3d","entry":"gelu_new","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"dc9ffc0f29e7fa3d"}},{"code_sha256_prefix":"2dda05eee1b6e0f7","entry":"image_grid","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/utils.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dda05eee1b6e0f7"}},{"code_sha256_prefix":"4b4a8984a2737ae5","entry":"inject_trainable_lora","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/lora.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/lora.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b4a8984a2737ae5"}},{"code_sha256_prefix":"3aec51ff694c19af","entry":"inject_trainable_lora_extended","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/lora.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/lora.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3aec51ff694c19af"}},{"code_sha256_prefix":"d793962f13865771","entry":"inpainting_dataloader","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/cli_lora_pti.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/cli_lora_pti.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d793962f13865771"}},{"code_sha256_prefix":"298713e11f263fda","entry":"prepare_clip_model_sets","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/utils.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"298713e11f263fda"}},{"code_sha256_prefix":"a1d506dd99d38923","entry":"reshape_weight_for_sd","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/to_ckpt_v2.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/to_ckpt_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1d506dd99d38923"}},{"code_sha256_prefix":"f999bed53b70c3ab","entry":"safe_open","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/safe_open.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/safe_open.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f999bed53b70c3ab"}},{"code_sha256_prefix":"ec4e915ba607e9c9","entry":"swin_ir_sr","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/preprocess_files.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/preprocess_files.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec4e915ba607e9c9"}},{"code_sha256_prefix":"bdf6d68955356de7","entry":"test_xformers_backwards","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/xformers_utils.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/xformers_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdf6d68955356de7"}},{"code_sha256_prefix":"aa63f591fb2e9bd7","entry":"text2img_dataloader","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/cli_lora_pti.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/cli_lora_pti.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa63f591fb2e9bd7"}},{"code_sha256_prefix":"8dbc9878c5043ebc","entry":"text_img_alignment","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/utils.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dbc9878c5043ebc"}},{"code_sha256_prefix":"cbef98786d899b3b","entry":"blip_captioning_dataset","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/preprocess_files.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/preprocess_files.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbef98786d899b3b"}},{"code_sha256_prefix":"98530605dba303f0","entry":"clipseg_mask_generator","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/preprocess_files.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/preprocess_files.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98530605dba303f0"}},{"code_sha256_prefix":"f2b2f19d941175af","entry":"convert_unet_state_dict","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/to_ckpt_v2.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/to_ckpt_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2b2f19d941175af"}},{"code_sha256_prefix":"effb6d8bd6b936e2","entry":"create_tensor","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/safe_open.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/safe_open.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"effb6d8bd6b936e2"}},{"code_sha256_prefix":"2f9880b526ff277e","entry":"extract_lora_ups_down","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/lora.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/lora.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f9880b526ff277e"}},{"code_sha256_prefix":"47566c65ffdc7fa2","entry":"lora_join","repo":"pilancilab/Riemannian_Preconditioned_LoRA","repo_kind":"official","path":"Object_Generation/lora_diffusion/lora_manager.py","file_url":"https://github.com/pilancilab/Riemannian_Preconditioned_LoRA/blob/HEAD/Object_Generation/lora_diffusion/lora_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47566c65ffdc7fa2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}