{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparsity-winning-twice-better-robust-1","title":"Sparsity Winning Twice: Better Robust Generalization from More Efficient Training","arxiv_id":"2202.09844","date":"2022-02-20","proceeding":"ICLR 2022 4","authors":["Tianlong Chen","Zhenyu Zhang","Pengjun Wang","Santosh Balachandra","Haoyu Ma","Zehao Wang","Zhangyang Wang"],"abstract":"Recent studies demonstrate that deep networks, even robustified by the state-of-the-art adversarial training (AT), still suffer from large robust generalization gaps, in addition to the much more expensive training costs than standard training. In this paper, we investigate this intriguing problem from a new perspective, i.e., injecting appropriate forms of sparsity during adversarial training. We introduce two alternatives for sparse adversarial training: (i) static sparsity, by leveraging recent results from the lottery ticket hypothesis to identify critical sparse subnetworks arising from the early training; (ii) dynamic sparsity, by allowing the sparse subnetwork to adaptively adjust its connectivity pattern (while sticking to the same sparsity ratio) throughout training. We find both static and dynamic sparse methods to yield win-win: substantially shrinking the robust generalization gap and alleviating the robust overfitting, meanwhile significantly saving training and inference FLOPs. Extensive experiments validate our proposals with multiple network architectures on diverse datasets, including CIFAR-10/100 and Tiny-ImageNet. For example, our methods reduce robust generalization gap and overfitting by 34.44% and 4.02%, with comparable robust/standard accuracy boosts and 87.83%/87.82% training/inference FLOPs savings on CIFAR-100 with ResNet-18. Besides, our approaches can be organically combined with existing regularizers, establishing new state-of-the-art results in AT. Codes are available in https://github.com/VITA-Group/Sparsity-Win-Robust-Generalization.","url_abs":"https://arxiv.org/abs/2202.09844v3","url_pdf":"https://arxiv.org/pdf/2202.09844v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparsity-winning-twice-better-robust-1","repo_url":"https://github.com/vita-group/sparsity-win-robust-generalization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2202.09844","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.09844"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vita-group/sparsity-win-robust-generalization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0beec913c324c82a","entry":"clamp","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"Robust-Bird/utils/function.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/Robust-Bird/utils/function.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0beec913c324c82a"}},{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"Robust-Bird/models/resnet.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/Robust-Bird/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"c1fae20306092b7b","entry":"cifar100_dataloaders","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"Robust-Bird/datasets.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/Robust-Bird/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1fae20306092b7b"}},{"code_sha256_prefix":"f46b4c2ca9177dd4","entry":"cifar10_dataloaders","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"Robust-Bird/datasets.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/Robust-Bird/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f46b4c2ca9177dd4"}},{"code_sha256_prefix":"7cb3feda876091c6","entry":"cifar10_test_dataloaders","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cb3feda876091c6"}},{"code_sha256_prefix":"c10449a1b159ff70","entry":"reshape_transform","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"grad_carm.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/grad_carm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c10449a1b159ff70"}},{"code_sha256_prefix":"235d303f6fbb9b91","entry":"resnet18","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"Robust-Bird/models/resnet.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/Robust-Bird/models/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"235d303f6fbb9b91"}},{"code_sha256_prefix":"ce067ba1b13d302e","entry":"tiny_imagenet_dataloaders","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"Robust-Bird/datasets.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/Robust-Bird/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce067ba1b13d302e"}},{"code_sha256_prefix":"a580b4aa6013d66c","entry":"train_epoch","repo":"vita-group/sparsity-win-robust-generalization","repo_kind":"official","path":"Robust-Bird/utils/function.py","file_url":"https://github.com/vita-group/sparsity-win-robust-generalization/blob/HEAD/Robust-Bird/utils/function.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a580b4aa6013d66c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}