{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/no-more-fine-tuning-an-experimental","title":"No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence","arxiv_id":"2207.11680","date":"2022-07-24","proceeding":null,"authors":["Chaozheng Wang","Yuanhang Yang","Cuiyun Gao","Yun Peng","Hongyu Zhang","Michael R. Lyu"],"abstract":"Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26\\% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.","url_abs":"https://arxiv.org/abs/2207.11680v1","url_pdf":"https://arxiv.org/pdf/2207.11680v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"no-more-fine-tuning-an-experimental","repo_url":"https://github.com/adf1178/pt4code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-summarization-1","task_name":"Code Summarization"},{"task_slug":"code-translation","task_name":"Code Translation"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"codebert","method_name":"CodeBERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2207.11680","atlas_url":"https://app.syntology.ai/?focus=2207.11680","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.11680"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adf1178/pt4code","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":14,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2e3e72e1e860afb2","entry":"add_lang_by_task","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/_utils.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e3e72e1e860afb2"}},{"code_sha256_prefix":"08ffc47da0ae6767","entry":"convert_clone_examples_to_features","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/_utils.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08ffc47da0ae6767"}},{"code_sha256_prefix":"70674cae33ae9c61","entry":"convert_examples_to_features","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/_utils.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70674cae33ae9c61"}},{"code_sha256_prefix":"835fca2953749b59","entry":"count_ngrams","repo":"adf1178/pt4code","repo_kind":"official","path":"summarization/bleu.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/summarization/bleu.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"835fca2953749b59"}},{"code_sha256_prefix":"7162c6b07c0d21f6","entry":"evaluate","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/run_defect.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/run_defect.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7162c6b07c0d21f6"}},{"code_sha256_prefix":"8cfe38019e9d6a49","entry":"get_model_size","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/models.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cfe38019e9d6a49"}},{"code_sha256_prefix":"90c23ee13a3e7488","entry":"read_examples","repo":"adf1178/pt4code","repo_kind":"official","path":"defect/finetune/finetune_t5.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/defect/finetune/finetune_t5.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90c23ee13a3e7488"}},{"code_sha256_prefix":"504a7d87c9b11731","entry":"add_args","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/configs.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/configs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"504a7d87c9b11731"}},{"code_sha256_prefix":"eb4f6d081f695b86","entry":"build_or_load_gen_model","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/models.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb4f6d081f695b86"}},{"code_sha256_prefix":"18886d59952fec97","entry":"eval_ppl_epoch","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/run_gen.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/run_gen.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18886d59952fec97"}},{"code_sha256_prefix":"a8459dd6372165af","entry":"evaluate","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/run_clone.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/run_clone.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8459dd6372165af"}},{"code_sha256_prefix":"4dfcf272e3b5535f","entry":"load_and_cache_clone_data","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/utils.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dfcf272e3b5535f"}},{"code_sha256_prefix":"db9fdc88ead105ab","entry":"load_and_cache_defect_data","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/utils.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db9fdc88ead105ab"}},{"code_sha256_prefix":"08e8e1fc7c5648f7","entry":"load_and_cache_gen_data","repo":"adf1178/pt4code","repo_kind":"official","path":"CodeT5-main/utils.py","file_url":"https://github.com/adf1178/pt4code/blob/HEAD/CodeT5-main/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08e8e1fc7c5648f7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}