{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-transferability-of-prompt-tuning-for","title":"On Transferability of Prompt Tuning for Natural Language Processing","arxiv_id":"2111.06719","date":"2021-11-12","proceeding":"NAACL 2022 7","authors":["Yusheng Su","Xiaozhi Wang","Yujia Qin","Chi-Min Chan","Yankai Lin","Huadong Wang","Kaiyue Wen","Zhiyuan Liu","Peng Li","Juanzi Li","Lei Hou","Maosong Sun","Jie zhou"],"abstract":"Prompt tuning (PT) is a promising parameter-efficient method to utilize extremely large pre-trained language models (PLMs), which can achieve comparable performance to full-parameter fine-tuning by only tuning a few soft prompts. However, PT requires much more training time than fine-tuning. Intuitively, knowledge transfer can help to improve the efficiency. To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work. We find that (1) in zero-shot setting, trained soft prompts can effectively transfer to similar tasks on the same PLM and also to other PLMs with a cross-model projector trained on similar tasks; (2) when used as initialization, trained soft prompts of similar tasks and projected prompts of other PLMs can significantly accelerate training and also improve the performance of PT. Moreover, to explore what decides prompt transferability, we investigate various transferability indicators and find that the overlapping rate of activated neurons strongly reflects the transferability, which suggests how the prompts stimulate PLMs is essential. Our findings show that prompt transfer is promising for improving PT, and further research shall focus more on prompts' stimulation to PLMs. The source code can be obtained from https://github.com/thunlp/Prompt-Transferability.","url_abs":"https://arxiv.org/abs/2111.06719v2","url_pdf":"https://arxiv.org/pdf/2111.06719v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-transferability-of-prompt-tuning-for","repo_url":"https://github.com/thunlp/Prompt-Transferability","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2111.06719","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.06719"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/Prompt-Transferability","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"74a65585ed82d97c","entry":"CosineSimilarity","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/caculate_metric_value_for_correlation.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/caculate_metric_value_for_correlation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74a65585ed82d97c"}},{"code_sha256_prefix":"fbd14dd3653a6903","entry":"EuclideanDistances","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/caculate_metric_value_for_correlation.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/caculate_metric_value_for_correlation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbd14dd3653a6903"}},{"code_sha256_prefix":"1b358a270966ccde","entry":"EuclideanDistances_per_token","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/caculate_metric_value_for_correlation.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/caculate_metric_value_for_correlation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b358a270966ccde"}},{"code_sha256_prefix":"2996dad43a48fbe2","entry":"acc","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/model/BinaryClsBERT.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/model/BinaryClsBERT.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2996dad43a48fbe2"}},{"code_sha256_prefix":"4e35da92886e6ffa","entry":"acc","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/model/PromptBert.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/model/PromptBert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e35da92886e6ffa"}},{"code_sha256_prefix":"b427b4ac98d3859e","entry":"cal_pearson","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/model/PromptBert.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/model/PromptBert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b427b4ac98d3859e"}},{"code_sha256_prefix":"b1a510c7c3fd60f4","entry":"pearson","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/model/PromptBert.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/model/PromptBert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a510c7c3fd60f4"}},{"code_sha256_prefix":"5a45143bbc682622","entry":"relu","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/activate_neuron_Roberta.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/activate_neuron_Roberta.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a45143bbc682622"}},{"code_sha256_prefix":"a8bc7776c327cfc2","entry":"topk","repo":"thunlp/Prompt-Transferability","repo_kind":"official","path":"Prompt-Transferability-1.0/activate_neuron_Roberta.py","file_url":"https://github.com/thunlp/Prompt-Transferability/blob/HEAD/Prompt-Transferability-1.0/activate_neuron_Roberta.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8bc7776c327cfc2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}