{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-compute-is-not-all-you-need-for","title":"Scaling Compute Is Not All You Need for Adversarial Robustness","arxiv_id":"2312.13131","date":"2023-12-20","proceeding":null,"authors":["Edoardo Debenedetti","Zishen Wan","Maksym Andriushchenko","Vikash Sehwag","Kshitij Bhardwaj","Bhavya Kailkhura"],"abstract":"The last six years have witnessed significant progress in adversarially robust deep learning. As evidenced by the CIFAR-10 dataset category in RobustBench benchmark, the accuracy under $\\ell_\\infty$ adversarial perturbations improved from 44\\% in \\citet{Madry2018Towards} to 71\\% in \\citet{peng2023robust}. Although impressive, existing state-of-the-art is still far from satisfactory. It is further observed that best-performing models are often very large models adversarially trained by industrial labs with significant computational budgets. In this paper, we aim to understand: ``how much longer can computing power drive adversarial robustness advances?\" To answer this question, we derive \\emph{scaling laws for adversarial robustness} which can be extrapolated in the future to provide an estimate of how much cost we would need to pay to reach a desired level of robustness. We show that increasing the FLOPs needed for adversarial training does not bring as much advantage as it does for standard training in terms of performance improvements. Moreover, we find that some of the top-performing techniques are difficult to exactly reproduce, suggesting that they are not robust enough for minor changes in the training setup. Our analysis also uncovers potentially worthwhile directions to pursue in future research. Finally, we make our benchmarking framework (built on top of \\texttt{timm}~\\citep{rw2019timm}) publicly available to facilitate future analysis in efficient robust deep learning.","url_abs":"https://arxiv.org/abs/2312.13131v1","url_pdf":"https://arxiv.org/pdf/2312.13131v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-compute-is-not-all-you-need-for","repo_url":"https://github.com/dedeswim/timm-adv-training","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"all","task_name":"All"},{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.13131","atlas_url":"https://app.syntology.ai/?focus=2312.13131","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.13131"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dedeswim/timm-adv-training","reach":{"status":"ok"}}],"summary":{"ran_honours":3,"ran":7,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":14,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"410bffb8a07b52e3","entry":"count_params","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/benchmark.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"410bffb8a07b52e3"}},{"code_sha256_prefix":"e35ff2bdc490fe54","entry":"check_end_of_training","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"check_end_of_training.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/check_end_of_training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e35ff2bdc490fe54"}},{"code_sha256_prefix":"dacaeb0776b0fadb","entry":"check_end_of_training","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"create_csv.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/create_csv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dacaeb0776b0fadb"}},{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"src/models/adv_resnet.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/src/models/adv_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"src/models/adv_resnet.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/src/models/adv_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"9ba2e71ff9efea7b","entry":"count_model_parameters","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"create_csv.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/create_csv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ba2e71ff9efea7b"}},{"code_sha256_prefix":"50d653ee6a59955c","entry":"cuda_timestamp","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/benchmark.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50d653ee6a59955c"}},{"code_sha256_prefix":"7784f1eaf0ecd721","entry":"float_or_none","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"create_csv.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/create_csv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7784f1eaf0ecd721"}},{"code_sha256_prefix":"3120bc13e5c3119a","entry":"init_linf","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"src/attacks.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/src/attacks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3120bc13e5c3119a"}},{"code_sha256_prefix":"d0c7744cdca9dcda","entry":"project_linf","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"src/attacks.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/src/attacks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0c7744cdca9dcda"}},{"code_sha256_prefix":"bdf711c61b603111","entry":"summary_row_dict","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"src/monitor.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/src/monitor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdf711c61b603111"}},{"code_sha256_prefix":"475160c5a1a1225a","entry":"timestamp","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/benchmark.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"475160c5a1a1225a"}},{"code_sha256_prefix":"a4a0d88d7ad50530","entry":"init_l2","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"src/attacks.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/src/attacks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4a0d88d7ad50530"}},{"code_sha256_prefix":"6e589cc8596033c9","entry":"parse_args","repo":"dedeswim/timm-adv-training","repo_kind":"official","path":"src/arg_parser.py","file_url":"https://github.com/dedeswim/timm-adv-training/blob/HEAD/src/arg_parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e589cc8596033c9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}