{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vb-lora-extreme-parameter-efficient-fine","title":"VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks","arxiv_id":"2405.15179","date":"2024-05-24","proceeding":null,"authors":["Yang Li","Shaobo Han","Shihao Ji"],"abstract":"As the adoption of large language models increases and the need for per-user or per-task model customization grows, the parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. To further reduce stored parameters, we introduce a \"divide-and-share\" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing parameters globally via a vector bank. As an instantiation of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable top-k admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available at https://github.com/leo-yangli/VB-LoRA. This method has been merged into the Hugging Face PEFT package.","url_abs":"https://arxiv.org/abs/2405.15179v3","url_pdf":"https://arxiv.org/pdf/2405.15179v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vb-lora-extreme-parameter-efficient-fine","repo_url":"https://github.com/leo-yangli/vb-lora","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.15179","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15179"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/leo-yangli/VB-LoRA","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leo-yangli/vb-lora","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran":9,"ran_draft_wrong":6,"unverified":1},"by_repo_kind":{"official":{"samples":17,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"0fe9cf6c00ef56ea","entry":"is_number","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/gsm8k_eval.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/gsm8k_eval.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe9cf6c00ef56ea"}},{"code_sha256_prefix":"e6de664c4943a7e0","entry":"VBLoRA","repo":"leo-yangli/vb-lora","repo_kind":"official","path":"NLU/vblora/vblora.py","file_url":"https://github.com/leo-yangli/vb-lora/blob/HEAD/NLU/vblora/vblora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6de664c4943a7e0"}},{"code_sha256_prefix":"66b3b5450182da3a","entry":"batch_data","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/MATH_eval.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/MATH_eval.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66b3b5450182da3a"}},{"code_sha256_prefix":"f82430123a91cf21","entry":"extract_unnatural_instructions_data","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"instruction_tuning/qlora.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/instruction_tuning/qlora.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f82430123a91cf21"}},{"code_sha256_prefix":"649fc48067a48529","entry":"find_all_linear_names","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/utils/merge_adapter_to_base_model.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/utils/merge_adapter_to_base_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"649fc48067a48529"}},{"code_sha256_prefix":"a4475703ff58ecf9","entry":"gelu_fast","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"NLU/NLU/src/transformers/activations.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/NLU/NLU/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4475703ff58ecf9"}},{"code_sha256_prefix":"45bb87451230d5e8","entry":"gelu_new","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"NLU/NLU/src/transformers/activations.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/NLU/NLU/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45bb87451230d5e8"}},{"code_sha256_prefix":"8ae8c583b21633b3","entry":"is_digit","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/grader.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/grader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ae8c583b21633b3"}},{"code_sha256_prefix":"e9a9686489ca49a9","entry":"last_boxed_only","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/utils.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9a9686489ca49a9"}},{"code_sha256_prefix":"90b5c896e5eaea5e","entry":"last_boxed_only_string","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/utils.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90b5c896e5eaea5e"}},{"code_sha256_prefix":"65c738adb08e913c","entry":"math_equal","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/grader.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/grader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65c738adb08e913c"}},{"code_sha256_prefix":"f63c83de2a26f67c","entry":"math_equal_process","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/grader.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/grader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f63c83de2a26f67c"}},{"code_sha256_prefix":"f889d53c68539013","entry":"mish","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"NLU/NLU/src/transformers/activations.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/NLU/NLU/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f889d53c68539013"}},{"code_sha256_prefix":"ec6403ff3372ffa2","entry":"preprocess","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/intruction_tuning_vblora.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/intruction_tuning_vblora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec6403ff3372ffa2"}},{"code_sha256_prefix":"f3bbe264b05aadd3","entry":"remove_boxed","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/MATH_eval.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/MATH_eval.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3bbe264b05aadd3"}},{"code_sha256_prefix":"86efa2d5fbd39bf6","entry":"train_tokenize_function","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/intruction_tuning_vblora.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/intruction_tuning_vblora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86efa2d5fbd39bf6"}},{"code_sha256_prefix":"0bbf876ced018c83","entry":"only_until_first_boxed_from_tokens","repo":"leo-yangli/VB-LoRA","repo_kind":"official","path":"math_instruction_tuning/instruction_tuning_eval/utils.py","file_url":"https://github.com/leo-yangli/VB-LoRA/blob/HEAD/math_instruction_tuning/instruction_tuning_eval/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bbf876ced018c83"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}