{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reliable-evaluation-of-adversarial-robustness","title":"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks","arxiv_id":"2003.01690","date":"2020-03-03","proceeding":"ICML 2020 1","authors":["Francesco Croce","Matthias Hein"],"abstract":"The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of adversarial defenses is often insufficient and thus gives a wrong impression of robustness. Many promising defenses could be broken later on, making it difficult to identify the state-of-the-art. Frequent pitfalls in the evaluation are improper tuning of hyperparameters of the attacks, gradient obfuscation or masking. In this paper we first propose two extensions of the PGD-attack overcoming failures due to suboptimal step size and problems of the objective function. We then combine our novel attacks with two complementary existing ones to form a parameter-free, computationally affordable and user-independent ensemble of attacks to test adversarial robustness. We apply our ensemble to over 50 models from papers published at recent top machine learning and computer vision venues. In all except one of the cases we achieve lower robust test accuracy than reported in these papers, often by more than $10\\%$, identifying several broken defenses.","url_abs":"https://arxiv.org/abs/2003.01690v2","url_pdf":"https://arxiv.org/pdf/2003.01690v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/fra31/auto-attack","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/max-andr/square-attack","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/alirezaabdollahpour/superdeepfool","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/jeromerony/adversarial-library","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/locuslab/robust_overfitting","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/pku-ml/rebat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/tuananhbui89/ASCL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"reliable-evaluation-of-adversarial-robustness","repo_url":"https://github.com/tuananhbui89/Crossing-Collaborative-Ensemble","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2003.01690","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2003.01690"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/BorealisAI/advertorch","reach":{"status":"ok","spdx":"LGPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fra31/auto-attack","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/max-andr/square-attack","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alirezaabdollahpour/superdeepfool","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tuananhbui89/Crossing-Collaborative-Ensemble","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/locuslab/robust_overfitting","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pku-ml/rebat","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jeromerony/adversarial-library","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tuananhbui89/ASCL","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":2,"unverified":2},"by_repo_kind":{"listed":{"samples":3,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"0832e3bf9440362b","entry":"generate_random_targets","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0832e3bf9440362b"}},{"code_sha256_prefix":"c6344b8a7b36aea8","entry":"get_all_targets","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"c6344b8a7b36aea8"}},{"code_sha256_prefix":"b20f1357b1d8dbf8","entry":"mixup_data","repo":"locuslab/robust_overfitting","repo_kind":"listed","path":"train_cifar.py","file_url":"https://github.com/locuslab/robust_overfitting/blob/HEAD/train_cifar.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b20f1357b1d8dbf8"}},{"code_sha256_prefix":"8a93e041134b597a","entry":"clamp","repo":"locuslab/robust_overfitting","repo_kind":"listed","path":"train_cifar.py","file_url":"https://github.com/locuslab/robust_overfitting/blob/HEAD/train_cifar.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a93e041134b597a"}},{"code_sha256_prefix":"fd670ddeddb80a69","entry":"init_lr_finder","repo":"jeromerony/adversarial-library","repo_kind":"listed","path":"adv_lib/attacks/augmented_lagrangian.py","file_url":"https://github.com/jeromerony/adversarial-library/blob/HEAD/adv_lib/attacks/augmented_lagrangian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fd670ddeddb80a69"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}