{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-comprehensive-evaluation-of-parameter","title":"A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Tasks","arxiv_id":"2312.15614","date":"2023-12-25","proceeding":null,"authors":["Wentao Zou","Qi Li","Jidong Ge","Chuanyi Li","Xiaoyu Shen","LiGuo Huang","Bin Luo"],"abstract":"Pre-trained models (PTMs) have achieved great success in various Software Engineering (SE) downstream tasks following the ``pre-train then fine-tune'' paradigm. As fully fine-tuning all parameters of PTMs can be computationally expensive, a widely used solution is parameter-efficient fine-tuning (PEFT), which freezes PTMs while introducing extra parameters. Though work has been done to test PEFT methods in the SE field, a comprehensive evaluation is still lacking. This paper aims to fill in this gap by evaluating the effectiveness of five PEFT methods on eight PTMs and four SE downstream tasks. For different tasks and PEFT methods, we seek answers to the following research questions: 1) Is it more effective to use PTMs trained specifically on source code, or is it sufficient to use PTMs trained on natural language text? 2) What is the impact of varying model sizes? 3) How does the model architecture affect the performance? Besides effectiveness, we also discuss the efficiency of PEFT methods, concerning the costs of required training time and GPU resource consumption. We hope that our findings can provide a deeper understanding of PEFT methods on various PTMs and SE downstream tasks. All the codes and data are available at \\url{https://github.com/zwtnju/PEFT.git}.","url_abs":"https://arxiv.org/abs/2312.15614v1","url_pdf":"https://arxiv.org/pdf/2312.15614v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-comprehensive-evaluation-of-parameter","repo_url":"https://github.com/zwtnju/peft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.15614","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.15614"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zwtnju/peft","reach":{"status":"ok"}}],"summary":{"ran":10,"unverified":4},"by_repo_kind":{"official":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"ccbc4540f0d52171","entry":"calculate_scores","repo":"zwtnju/peft","repo_kind":"official","path":"clone/evaluator/evaluator.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/clone/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ccbc4540f0d52171"}},{"code_sha256_prefix":"41d07eb6c8fa7fe3","entry":"calculate_scores","repo":"zwtnju/peft","repo_kind":"official","path":"defect/evaluator/evaluator.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/defect/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41d07eb6c8fa7fe3"}},{"code_sha256_prefix":"68f5c0b24ac09348","entry":"convert_examples_to_features","repo":"zwtnju/peft","repo_kind":"official","path":"clone/code/clone.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/clone/code/clone.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68f5c0b24ac09348"}},{"code_sha256_prefix":"b4f0a16045f53d0b","entry":"convert_examples_to_features","repo":"zwtnju/peft","repo_kind":"official","path":"defect/code/defect.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/defect/code/defect.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4f0a16045f53d0b"}},{"code_sha256_prefix":"44937ddb7444bf3f","entry":"convert_examples_to_features","repo":"zwtnju/peft","repo_kind":"official","path":"search/code/search.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/search/code/search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44937ddb7444bf3f"}},{"code_sha256_prefix":"6244f710493c123f","entry":"get_example","repo":"zwtnju/peft","repo_kind":"official","path":"clone/code/clone.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/clone/code/clone.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6244f710493c123f"}},{"code_sha256_prefix":"73ade802c00b1275","entry":"get_model_size","repo":"zwtnju/peft","repo_kind":"official","path":"clone/code/utils.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/clone/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73ade802c00b1275"}},{"code_sha256_prefix":"fef237b7756f46a2","entry":"read_answers","repo":"zwtnju/peft","repo_kind":"official","path":"clone/evaluator/evaluator.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/clone/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fef237b7756f46a2"}},{"code_sha256_prefix":"bc0a52fa08b13927","entry":"read_predictions","repo":"zwtnju/peft","repo_kind":"official","path":"clone/evaluator/evaluator.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/clone/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc0a52fa08b13927"}},{"code_sha256_prefix":"c5842e6ad19a6199","entry":"read_predictions","repo":"zwtnju/peft","repo_kind":"official","path":"defect/evaluator/evaluator.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/defect/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5842e6ad19a6199"}},{"code_sha256_prefix":"36c97f6df416b811","entry":"evaluate","repo":"zwtnju/peft","repo_kind":"official","path":"defect/code/defect.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/defect/code/defect.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36c97f6df416b811"}},{"code_sha256_prefix":"a057595416cbd19e","entry":"evaluate","repo":"zwtnju/peft","repo_kind":"official","path":"search/code/search.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/search/code/search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a057595416cbd19e"}},{"code_sha256_prefix":"c931ca2e34e6dbd9","entry":"load_and_cache_examples","repo":"zwtnju/peft","repo_kind":"official","path":"clone/code/clone.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/clone/code/clone.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c931ca2e34e6dbd9"}},{"code_sha256_prefix":"5544b8be3dc13963","entry":"read_answers","repo":"zwtnju/peft","repo_kind":"official","path":"defect/evaluator/evaluator.py","file_url":"https://github.com/zwtnju/peft/blob/HEAD/defect/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5544b8be3dc13963"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}