{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-semi-supervised-learning-with","title":"Rethinking Semi-supervised Learning with Language Models","arxiv_id":"2305.13002","date":"2023-05-22","proceeding":null,"authors":["Zhengxiang Shi","Francesco Tonolini","Nikolaos Aletras","Emine Yilmaz","Gabriella Kazai","Yunlong Jiao"],"abstract":"Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks. Currently, there are two popular approaches to make use of unlabelled data: Self-training (ST) and Task-adaptive pre-training (TAPT). ST uses a teacher model to assign pseudo-labels to the unlabelled data, while TAPT continues pre-training on the unlabelled data before fine-tuning. To the best of our knowledge, the effectiveness of TAPT in SSL tasks has not been systematically studied, and no previous work has directly compared TAPT and ST in terms of their ability to utilize the pool of unlabelled data. In this paper, we provide an extensive empirical study comparing five state-of-the-art ST approaches and TAPT across various NLP tasks and data sizes, including in- and out-of-domain settings. Surprisingly, we find that TAPT is a strong and more robust SSL learner, even when using just a few hundred unlabelled samples or in the presence of domain shifts, compared to more sophisticated ST approaches, and tends to bring greater improvements in SSL than in fully-supervised settings. Our further analysis demonstrates the risks of using ST approaches when the size of labelled or unlabelled data is small or when domain shifts exist. We offer a fresh perspective for future SSL research, suggesting the use of unsupervised pre-training objectives over dependency on pseudo labels.","url_abs":"https://arxiv.org/abs/2305.13002v1","url_pdf":"https://arxiv.org/pdf/2305.13002v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-semi-supervised-learning-with","repo_url":"https://github.com/amzn/pretraining-or-self-training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"rethinking-semi-supervised-learning-with","repo_url":"https://github.com/zhengxiangshi/powerfulpromptft","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"pseudo-label","task_name":"Pseudo Label"},{"task_slug":"semi-supervised-text-classification-1","task_name":"Semi-Supervised Text Classification"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.13002","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.13002"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhengxiangshi/powerfulpromptft","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amzn/pretraining-or-self-training","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3346c66c873fff6a","entry":"comatch_contrastive_loss","repo":"amzn/pretraining-or-self-training","repo_kind":"official","path":"semilearn/algorithms/comatch/comatch.py","file_url":"https://github.com/amzn/pretraining-or-self-training/blob/HEAD/semilearn/algorithms/comatch/comatch.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3346c66c873fff6a"}},{"code_sha256_prefix":"88f4962784e09b88","entry":"param_groups_weight_decay","repo":"amzn/pretraining-or-self-training","repo_kind":"official","path":"semilearn/nets/utils.py","file_url":"https://github.com/amzn/pretraining-or-self-training/blob/HEAD/semilearn/nets/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88f4962784e09b88"}},{"code_sha256_prefix":"4c0668013863ad3f","entry":"resize_pos_embed_vit","repo":"amzn/pretraining-or-self-training","repo_kind":"official","path":"semilearn/nets/utils.py","file_url":"https://github.com/amzn/pretraining-or-self-training/blob/HEAD/semilearn/nets/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c0668013863ad3f"}},{"code_sha256_prefix":"4331f5778c460265","entry":"rotate_img","repo":"amzn/pretraining-or-self-training","repo_kind":"official","path":"semilearn/algorithms/crmatch/crmatch.py","file_url":"https://github.com/amzn/pretraining-or-self-training/blob/HEAD/semilearn/algorithms/crmatch/crmatch.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4331f5778c460265"}},{"code_sha256_prefix":"b158ea331635a6c9","entry":"load_checkpoint","repo":"amzn/pretraining-or-self-training","repo_kind":"official","path":"semilearn/nets/utils.py","file_url":"https://github.com/amzn/pretraining-or-self-training/blob/HEAD/semilearn/nets/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b158ea331635a6c9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}