{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-robustness-the-cost-of-accuracy-a","title":"Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Models","arxiv_id":"1808.01688","date":"2018-08-05","proceeding":"ECCV 2018 9","authors":["Dong Su","huan zhang","Hongge Chen","Jin-Feng Yi","Pin-Yu Chen","Yupeng Gao"],"abstract":"The prediction accuracy has been the long-lasting and sole standard for\ncomparing the performance of different image classification models, including\nthe ImageNet competition. However, recent studies have highlighted the lack of\nrobustness in well-trained deep neural networks to adversarial examples.\nVisually imperceptible perturbations to natural images can easily be crafted\nand mislead the image classifiers towards misclassification. To demystify the\ntrade-offs between robustness and accuracy, in this paper we thoroughly\nbenchmark 18 ImageNet models using multiple robustness metrics, including the\ndistortion, success rate and transferability of adversarial examples between\n306 pairs of models. Our extensive experimental results reveal several new\ninsights: (1) linear scaling law - the empirical $\\ell_2$ and $\\ell_\\infty$\ndistortion metrics scale linearly with the logarithm of classification error;\n(2) model architecture is a more critical factor to robustness than model size,\nand the disclosed accuracy-robustness Pareto frontier can be used as an\nevaluation criterion for ImageNet model designers; (3) for a similar network\narchitecture, increasing network depth slightly improves robustness in\n$\\ell_\\infty$ distortion; (4) there exist models (in VGG family) that exhibit\nhigh adversarial transferability, while most adversarial examples crafted from\none model can only be transferred within the same family. Experiment code is\npublicly available at \\url{https://github.com/huanzhang12/Adversarial_Survey}.","url_abs":"http://arxiv.org/abs/1808.01688v2","url_pdf":"http://arxiv.org/pdf/1808.01688v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-robustness-the-cost-of-accuracy-a","repo_url":"https://github.com/huanzhang12/Adversarial_Survey","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"is-robustness-the-cost-of-accuracy-a","repo_url":"https://github.com/IBM/ImageNet-Robustness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1808.01688","atlas_url":"https://app.syntology.ai/?focus=1808.01688","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1808.01688"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huanzhang12/Adversarial_Survey","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/IBM/ImageNet-Robustness","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2ff117f31964139f","entry":"keep_aspect_ratio_transform","repo":"IBM/ImageNet-Robustness","repo_kind":"listed","path":"setup_imagenet.py","file_url":"https://github.com/IBM/ImageNet-Robustness/blob/HEAD/setup_imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ff117f31964139f"}},{"code_sha256_prefix":"e15a35e7dfc912b0","entry":"readimg","repo":"IBM/ImageNet-Robustness","repo_kind":"listed","path":"setup_imagenet.py","file_url":"https://github.com/IBM/ImageNet-Robustness/blob/HEAD/setup_imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e15a35e7dfc912b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}