{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/don-t-stop-pretraining-make-prompt-based-fine","title":"Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner","arxiv_id":"2305.01711","date":"2023-05-02","proceeding":null,"authors":["Zhengxiang Shi","Aldo Lipani"],"abstract":"Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP). In this study, we re-visit the widely accepted notion in NLP that continued pre-training LMs on task-related texts improves the performance of fine-tuning (FT) in downstream tasks. Through experiments on eight single-sentence tasks and eight sentence-pair tasks in both semi-supervised and fully-supervised settings, we find that conventional continued pre-training does not consistently provide benefits and can even be detrimental for sentence-pair tasks or when prompt-based FT is used. To tackle these issues, we propose Prompt-based Continued Pre-training (PCP), which combines the idea of instruction tuning with conventional continued pre-training. Our approach aims to improve the performance of prompt-based FT by presenting both task-related texts and prompt templates to LMs through unsupervised pre-training objectives before fine-tuning for the target task. Our empirical evaluations on 21 benchmarks demonstrate that the PCP consistently improves the performance of state-of-the-art prompt-based FT approaches (up to 20.1% absolute) in both semi-supervised and fully-supervised settings, even with only hundreds of unlabelled examples. Additionally, prompt-based FT with the PCP outperforms state-of-the-art semi-supervised approaches with greater simplicity, eliminating the need for an iterative process and extra data augmentation. Our further analysis explores the performance lower bound of the PCP and reveals that the advantages of PCP persist across different sizes of models and datasets.","url_abs":"https://arxiv.org/abs/2305.01711v4","url_pdf":"https://arxiv.org/pdf/2305.01711v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"don-t-stop-pretraining-make-prompt-based-fine","repo_url":"https://github.com/zhengxiangshi/powerfulpromptft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"don-t-stop-pretraining-make-prompt-based-fine","repo_url":"https://github.com/zhengxiangshi/dept","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.01711","atlas_url":"https://app.syntology.ai/?focus=2305.01711","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.01711"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZhengxiangShi/PowerfulPromptFT","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhengxiangshi/dept","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhengxiangshi/powerfulpromptft","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5f5e5de530a5a8b4","entry":"count_parameters","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/model.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f5e5de530a5a8b4"}},{"code_sha256_prefix":"ff1816cf44827a08","entry":"dart_template","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/convert_to_pretrain_pcp.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/convert_to_pretrain_pcp.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff1816cf44827a08"}},{"code_sha256_prefix":"d9f08b1debac0613","entry":"dart_ts_template","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/convert_to_pretrain_pcp.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/convert_to_pretrain_pcp.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9f08b1debac0613"}},{"code_sha256_prefix":"126ae8eeb63d59bf","entry":"get_label","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/convert_to_pretrain_tapt_1.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/convert_to_pretrain_tapt_1.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"126ae8eeb63d59bf"}},{"code_sha256_prefix":"8e27652a29c8adcc","entry":"load_datasets","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/convert_to_pretrain_tapt_1.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/convert_to_pretrain_tapt_1.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e27652a29c8adcc"}},{"code_sha256_prefix":"efd1b7718b950470","entry":"split_header","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/convert_to_pretrain_tapt_1.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/convert_to_pretrain_tapt_1.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efd1b7718b950470"}},{"code_sha256_prefix":"edb11b8546838e1f","entry":"ssl_template","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/convert_to_pretrain_pcp.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/convert_to_pretrain_pcp.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edb11b8546838e1f"}},{"code_sha256_prefix":"b62076c5483738dc","entry":"text_classification_metrics","repo":"ZhengxiangShi/PowerfulPromptFT","repo_kind":"official","path":"src/processors.py","file_url":"https://github.com/ZhengxiangShi/PowerfulPromptFT/blob/HEAD/src/processors.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b62076c5483738dc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}