{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/drawing-robust-scratch-tickets-subnetworks","title":"Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks","arxiv_id":"2110.14068","date":"2021-10-26","proceeding":"NeurIPS 2021 12","authors":["Yonggan Fu","Qixuan Yu","Yang Zhang","Shang Wu","Xu Ouyang","David Cox","Yingyan Celine Lin"],"abstract":"Deep Neural Networks (DNNs) are known to be vulnerable to adversarial attacks, i.e., an imperceptible perturbation to the input can mislead DNNs trained on clean images into making erroneous predictions. To tackle this, adversarial training is currently the most effective defense method, by augmenting the training set with adversarial samples generated on the fly. Interestingly, we discover for the first time that there exist subnetworks with inborn robustness, matching or surpassing the robust accuracy of the adversarially trained networks with comparable model sizes, within randomly initialized networks without any model training, indicating that adversarial training on model weights is not indispensable towards adversarial robustness. We name such subnetworks Robust Scratch Tickets (RSTs), which are also by nature efficient. Distinct from the popular lottery ticket hypothesis, neither the original dense networks nor the identified RSTs need to be trained. To validate and understand this fascinating finding, we further conduct extensive experiments to study the existence and properties of RSTs under different models, datasets, sparsity patterns, and attacks, drawing insights regarding the relationship between DNNs' robustness and their initialization/overparameterization. Furthermore, we identify the poor adversarial transferability between RSTs of different sparsity ratios drawn from the same randomly initialized dense network, and propose a Random RST Switch (R2S) technique, which randomly switches between different RSTs, as a novel defense method built on top of RSTs. We believe our findings about RSTs have opened up a new perspective to study model robustness and extend the lottery ticket hypothesis.","url_abs":"https://arxiv.org/abs/2110.14068v4","url_pdf":"https://arxiv.org/pdf/2110.14068v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"drawing-robust-scratch-tickets-subnetworks","repo_url":"https://github.com/rice-eic/robust-scratch-ticket","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.14068","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.14068"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rice-eic/robust-scratch-ticket","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8a42a6951b0c5e57","entry":"accumulate","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"utils/net_utils.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a42a6951b0c5e57"}},{"code_sha256_prefix":"177a4abf40724f6e","entry":"arg_to_varname","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"config_rst/parser.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/config_rst/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"177a4abf40724f6e"}},{"code_sha256_prefix":"ffc9d7cced6f71fb","entry":"argv_to_vars","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"config_rst/parser.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/config_rst/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffc9d7cced6f71fb"}},{"code_sha256_prefix":"8a93e041134b597a","entry":"clamp","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"trainers/default.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/trainers/default.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a93e041134b597a"}},{"code_sha256_prefix":"6071f5c0529621f1","entry":"fgsm","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"trainers/default.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/trainers/default.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6071f5c0529621f1"}},{"code_sha256_prefix":"717f5458343a0b63","entry":"get_lr","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"utils/net_utils.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"717f5458343a0b63"}},{"code_sha256_prefix":"9fd69d0fcf184abd","entry":"set_gpu","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"train_full.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/train_full.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fd69d0fcf184abd"}},{"code_sha256_prefix":"a99f170ebd682af5","entry":"trim_preceding_hyphens","repo":"RICE-EIC/Robust-Scratch-Ticket","repo_kind":"official","path":"config_rst/parser.py","file_url":"https://github.com/RICE-EIC/Robust-Scratch-Ticket/blob/HEAD/config_rst/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a99f170ebd682af5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}