{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/train-no-evil-selective-masking-for-task","title":"Train No Evil: Selective Masking for Task-Guided Pre-Training","arxiv_id":"2004.09733","date":"2020-04-21","proceeding":"EMNLP 2020 11","authors":["Yuxian Gu","Zhengyan Zhang","Xiaozhi Wang","Zhiyuan Liu","Maosong Sun"],"abstract":"Recently, pre-trained language models mostly follow the pre-train-then-fine-tuning paradigm and have achieved great performance on various downstream tasks. However, since the pre-training stage is typically task-agnostic and the fine-tuning stage usually suffers from insufficient supervised data, the models cannot always well capture the domain-specific and task-specific patterns. In this paper, we propose a three-stage framework by adding a task-guided pre-training stage with selective masking between general pre-training and fine-tuning. In this stage, the model is trained by masked language modeling on in-domain unsupervised data to learn domain-specific patterns and we propose a novel selective masking strategy to learn task-specific patterns. Specifically, we design a method to measure the importance of each token in sequences and selectively mask the important tokens. Experimental results on two sentiment analysis tasks show that our method can achieve comparable or even better performance with less than 50% of computation cost, which indicates our method is both effective and efficient. The source code of this paper can be obtained from https://github.com/thunlp/SelectiveMasking.","url_abs":"https://arxiv.org/abs/2004.09733v2","url_pdf":"https://arxiv.org/pdf/2004.09733v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"train-no-evil-selective-masking-for-task","repo_url":"https://github.com/thunlp/SelectiveMasking","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.09733","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.09733"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/SelectiveMasking","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"f9df374724e4186c","entry":"InputExample","repo":"thunlp/SelectiveMasking","repo_kind":"official","path":"mask_model_pretrain.py","file_url":"https://github.com/thunlp/SelectiveMasking/blob/HEAD/mask_model_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9df374724e4186c"}},{"code_sha256_prefix":"e4c84991f65ae5cf","entry":"readfile","repo":"thunlp/SelectiveMasking","repo_kind":"official","path":"mask_model_pretrain.py","file_url":"https://github.com/thunlp/SelectiveMasking/blob/HEAD/mask_model_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4c84991f65ae5cf"}},{"code_sha256_prefix":"20222d5099122093","entry":"DataProcessor","repo":"thunlp/SelectiveMasking","repo_kind":"official","path":"mask_model_pretrain.py","file_url":"https://github.com/thunlp/SelectiveMasking/blob/HEAD/mask_model_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20222d5099122093"}},{"code_sha256_prefix":"28a573663476e0bc","entry":"MaskGenProcessor","repo":"thunlp/SelectiveMasking","repo_kind":"official","path":"mask_model_pretrain.py","file_url":"https://github.com/thunlp/SelectiveMasking/blob/HEAD/mask_model_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28a573663476e0bc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}