{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/increasing-model-capacity-for-free-a-simple","title":"Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning","arxiv_id":"2407.01320","date":"2024-07-01","proceeding":null,"authors":["Haobo Song","Hao Zhao","Soumajit Majumder","Tao Lin"],"abstract":"Fine-tuning large pre-trained foundation models, such as the 175B GPT-3, has attracted more attention for downstream tasks recently. While parameter-efficient fine-tuning methods have been proposed and proven effective without retraining all model parameters, their performance is limited by the capacity of incremental modules, especially under constrained parameter budgets. \\\\ To overcome this challenge, we propose CapaBoost, a simple yet effective strategy that enhances model capacity by leveraging low-rank updates through parallel weight modules in target layers. By applying static random masks to the shared weight matrix, CapaBoost constructs a diverse set of weight matrices, effectively increasing the rank of incremental weights without adding parameters. Notably, our approach can be seamlessly integrated into various existing parameter-efficient fine-tuning methods. We extensively validate the efficacy of CapaBoost through experiments on diverse downstream tasks, including natural language understanding, question answering, and image classification. Our results demonstrate significant improvements over baselines, without incurring additional computation or storage costs. Our code is available at \\url{https://github.com/LINs-lab/CapaBoost}.","url_abs":"https://arxiv.org/abs/2407.01320v1","url_pdf":"https://arxiv.org/pdf/2407.01320v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"increasing-model-capacity-for-free-a-simple","repo_url":"https://github.com/lins-lab/capaboost","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.01320","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.01320"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LINs-lab/CapaBoost","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lins-lab/capaboost","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a727df5fd2f5599a","entry":"check_number_comma","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/convert_slow_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a727df5fd2f5599a"}},{"code_sha256_prefix":"d770e1dd1d799a47","entry":"ensure_valid_input","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d770e1dd1d799a47"}},{"code_sha256_prefix":"7efe3db5c9e2f94d","entry":"generate_identified_filename","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7efe3db5c9e2f94d"}},{"code_sha256_prefix":"54fa343f001c7cff","entry":"kronecker_product","repo":"lins-lab/capaboost","repo_kind":"official","path":"src/transformers/adapters/modeling.py","file_url":"https://github.com/lins-lab/capaboost/blob/HEAD/src/transformers/adapters/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54fa343f001c7cff"}},{"code_sha256_prefix":"561495a2688b60b1","entry":"convert_slow_tokenizer","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/convert_slow_tokenizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"561495a2688b60b1"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"dd3166a87bc972ff","entry":"get_configuration_file","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/configuration_utils.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/configuration_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd3166a87bc972ff"}},{"code_sha256_prefix":"0da835817b8a57bc","entry":"infer_shapes","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0da835817b8a57bc"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"LINs-lab/CapaBoost","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/LINs-lab/CapaBoost/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}