{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dora-weight-decomposed-low-rank-adaptation","title":"DoRA: Weight-Decomposed Low-Rank Adaptation","arxiv_id":"2402.09353","date":"2024-02-14","proceeding":null,"authors":["Shih-Yang Liu","Chien-Yi Wang","Hongxu Yin","Pavlo Molchanov","Yu-Chiang Frank Wang","Kwang-Ting Cheng","Min-Hung Chen"],"abstract":"Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing \\ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. \\ours~consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code is available at https://github.com/NVlabs/DoRA.","url_abs":"https://arxiv.org/abs/2402.09353v6","url_pdf":"https://arxiv.org/pdf/2402.09353v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dora-weight-decomposed-low-rank-adaptation","repo_url":"https://github.com/NVlabs/DoRA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"dora-weight-decomposed-low-rank-adaptation","repo_url":"https://github.com/nbasyl/DoRA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"dora-weight-decomposed-low-rank-adaptation","repo_url":"https://github.com/catid/dora","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"dora-weight-decomposed-low-rank-adaptation","repo_url":"https://github.com/seanzhang-zhichen/llama3-chinese","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dora-weight-decomposed-low-rank-adaptation","repo_url":"https://github.com/ayyucedemirbas/DoRA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/parameter-efficient-fine-tuning-on-boolq","task":"parameter-efficient fine-tuning","dataset":"BoolQ","model":"LLaMA2-7b","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy (% )":"81.93"},"uses_additional_data":false},{"leaderboard":"/sota/parameter-efficient-fine-tuning-on-hellaswag","task":"parameter-efficient fine-tuning","dataset":"HellaSwag","model":"LLaMA2-7b","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy (% )":"76.27"},"uses_additional_data":true},{"leaderboard":"/sota/parameter-efficient-fine-tuning-on-winogrande","task":"parameter-efficient fine-tuning","dataset":"WinoGrande","model":"LLaMA2-7b","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy (% )":"70.09"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.09353","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.09353"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVlabs/DoRA","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/catid/dora","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nbasyl/DoRA","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seanzhang-zhichen/llama3-chinese","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ayyucedemirbas/DoRA","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":6,"unverified":7},"by_repo_kind":{"official":{"samples":14,"ran":7,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"68e41c28a6dfe81d","entry":"collate_fn","repo":"NVlabs/DoRA","repo_kind":"official","path":"image_video_text_understanding/feature_extraction/refcocog_mattnet.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/image_video_text_understanding/feature_extraction/refcocog_mattnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68e41c28a6dfe81d"}},{"code_sha256_prefix":"2dc17f5e6fb39b3f","entry":"evaluate","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/multi_dataset_eval.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/multi_dataset_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2dc17f5e6fb39b3f"}},{"code_sha256_prefix":"f040823d10747c45","entry":"generate_data","repo":"catid/dora","repo_kind":"listed","path":"dora.py","file_url":"https://github.com/catid/dora/blob/HEAD/dora.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f040823d10747c45"}},{"code_sha256_prefix":"56e674752db00649","entry":"generate_prompt","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/finetune.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/finetune.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56e674752db00649"}},{"code_sha256_prefix":"0dbc7fc6c518d0fb","entry":"generate_prompt","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/generate.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/generate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dbc7fc6c518d0fb"}},{"code_sha256_prefix":"1c6b8be890b6118a","entry":"generate_prompt","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/commonsense_evaluate.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/commonsense_evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c6b8be890b6118a"}},{"code_sha256_prefix":"56a9466ff35b35bb","entry":"load_data","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/commonsense_evaluate.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/commonsense_evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56a9466ff35b35bb"}},{"code_sha256_prefix":"a4f5d0414a1cf9c6","entry":"translate_state_dict_key","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/export_state_dict_checkpoint.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/export_state_dict_checkpoint.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4f5d0414a1cf9c6"}},{"code_sha256_prefix":"0d95286fcf19dd47","entry":"get_peft_state_maybe_zero_3","repo":"NVlabs/DoRA","repo_kind":"official","path":"visual_instruction_tuning/llava/train/train_dora.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/visual_instruction_tuning/llava/train/train_dora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0d95286fcf19dd47"}},{"code_sha256_prefix":"293ec4b71aa63222","entry":"get_peft_state_non_dora_maybe_zero_3","repo":"NVlabs/DoRA","repo_kind":"official","path":"visual_instruction_tuning/llava/train/train_dora.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/visual_instruction_tuning/llava/train/train_dora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"293ec4b71aa63222"}},{"code_sha256_prefix":"415f46d8a341e28d","entry":"main","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/commonsense_evaluate.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/commonsense_evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"415f46d8a341e28d"}},{"code_sha256_prefix":"3398969fc7af6ff7","entry":"main","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/evaluate.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3398969fc7af6ff7"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"NVlabs/DoRA","repo_kind":"official","path":"visual_instruction_tuning/llava/train/train_dora.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/visual_instruction_tuning/llava/train/train_dora.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}},{"code_sha256_prefix":"a2f49e13f6e13d6b","entry":"permute","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/export_state_dict_checkpoint.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/export_state_dict_checkpoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2f49e13f6e13d6b"}},{"code_sha256_prefix":"6e275f232bbfb9b6","entry":"unpermute","repo":"NVlabs/DoRA","repo_kind":"official","path":"commonsense_reasoning/export_state_dict_checkpoint.py","file_url":"https://github.com/NVlabs/DoRA/blob/HEAD/commonsense_reasoning/export_state_dict_checkpoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e275f232bbfb9b6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}