{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/super-tickets-in-pre-trained-language-models","title":"Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization","arxiv_id":"2105.12002","date":"2021-05-25","proceeding":"ACL 2021 5","authors":["Chen Liang","Simiao Zuo","Minshuo Chen","Haoming Jiang","Xiaodong Liu","Pengcheng He","Tuo Zhao","Weizhu Chen"],"abstract":"The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can match the performance of the full model. In this paper, we study such a collection of tickets, which is referred to as ``winning tickets'', in extremely over-parametrized models, e.g., pre-trained language models. We observe that at certain compression ratios, the generalization performance of the winning tickets can not only match but also exceed that of the full model. In particular, we observe a phase transition phenomenon: As the compression ratio increases, generalization performance of the winning tickets first improves then deteriorates after a certain threshold. We refer to the tickets on the threshold as ``super tickets''. We further show that the phase transition is task and model dependent -- as the model size becomes larger and the training data set becomes smaller, the transition becomes more pronounced. Our experiments on the GLUE benchmark show that the super tickets improve single task fine-tuning by $0.9$ points on BERT-base and $1.0$ points on BERT-large, in terms of task-average score. We also demonstrate that adaptively sharing the super tickets across tasks benefits multi-task learning.","url_abs":"https://arxiv.org/abs/2105.12002v2","url_pdf":"https://arxiv.org/pdf/2105.12002v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"super-tickets-in-pre-trained-language-models","repo_url":"https://github.com/cliang1453/super-structured-lottery-tickets","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.12002","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.12002"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/namisan/mt-dnn","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cliang1453/super-structured-lottery-tickets","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":4,"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":15,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"211753ba29188d4e","entry":"gelu","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"module/modeling_bert.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/module/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"211753ba29188d4e"}},{"code_sha256_prefix":"77601724cad03f95","entry":"gelu_new","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"module/modeling_bert.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/module/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"77601724cad03f95"}},{"code_sha256_prefix":"691b566282a42d88","entry":"linear","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"module/common.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/module/common.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"691b566282a42d88"}},{"code_sha256_prefix":"0c145728a35789fa","entry":"load_tf_weights_in_bert","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"module/modeling_bert.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/module/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"0c145728a35789fa"}},{"code_sha256_prefix":"dea985d5f4e928e8","entry":"warmup_linear_xdl","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"module/bert_optim.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/module/bert_optim.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dea985d5f4e928e8"}},{"code_sha256_prefix":"1294e1fa682e2b8c","entry":"bertgelu","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"module/common.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/module/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1294e1fa682e2b8c"}},{"code_sha256_prefix":"a034fa2c801b129f","entry":"build_data_single","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"extractor.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a034fa2c801b129f"}},{"code_sha256_prefix":"3764f415f19251f4","entry":"determine_masking_sequence","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"masking.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/masking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"3764f415f19251f4"}},{"code_sha256_prefix":"0de7059b7a331eb0","entry":"encode_pieces","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"data_utils/xlnet_utils.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/data_utils/xlnet_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"0de7059b7a331eb0"}},{"code_sha256_prefix":"273950f7bcb961ae","entry":"load_data","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"extractor.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"273950f7bcb961ae"}},{"code_sha256_prefix":"b19697c998adb1ea","entry":"preprocess_text","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"data_utils/xlnet_utils.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/data_utils/xlnet_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b19697c998adb1ea"}},{"code_sha256_prefix":"0e5615f8994003cf","entry":"printable_text","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"data_utils/xlnet_utils.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/data_utils/xlnet_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"0e5615f8994003cf"}},{"code_sha256_prefix":"47cf8803722c3336","entry":"swish","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"module/common.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/module/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"47cf8803722c3336"}},{"code_sha256_prefix":"fca2c0b9e35c90c3","entry":"what_to_mask","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"masking.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/masking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"fca2c0b9e35c90c3"}},{"code_sha256_prefix":"1218c6ee351ccc7f","entry":"what_to_threshold","repo":"cliang1453/super-structured-lottery-tickets","repo_kind":"official","path":"masking.py","file_url":"https://github.com/cliang1453/super-structured-lottery-tickets/blob/HEAD/masking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1218c6ee351ccc7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}