{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/smooth-adversarial-training","title":"Smooth Adversarial Training","arxiv_id":"2006.14536","date":"2020-06-25","proceeding":null,"authors":["Cihang Xie","Mingxing Tan","Boqing Gong","Alan Yuille","Quoc V. Le"],"abstract":"It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for \"free\", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://github.com/cihangxie/SmoothAdversarialTraining.","url_abs":"https://arxiv.org/abs/2006.14536v2","url_pdf":"https://arxiv.org/pdf/2006.14536v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"smooth-adversarial-training","repo_url":"https://github.com/cihangxie/SmoothAdversarialTraining","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-defense","task_name":"Adversarial Defense"},{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/adversarial-defense-on-imagenet-non-targeted","task":"Adversarial Defense","dataset":"ImageNet (non-targeted PGD, max perturbation=4)","model":"SAT-EfficientNet-L1","rank_in_archive_order":1,"of":5,"metrics":{"Accuracy":"58.6%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.14536","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.14536"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cihangxie/SmoothAdversarialTraining","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":13},"by_repo_kind":{"official":{"samples":14,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2f023d4690a70195","entry":"solarize","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/autoaugment.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/autoaugment.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f023d4690a70195"}},{"code_sha256_prefix":"2a1ec1fdff9feda1","entry":"blend","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/autoaugment.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/autoaugment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a1ec1fdff9feda1"}},{"code_sha256_prefix":"9f8ac1471e474b31","entry":"build_learning_rate","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/utils.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f8ac1471e474b31"}},{"code_sha256_prefix":"8b0555eb0514ca06","entry":"build_optimizer","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/utils.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b0555eb0514ca06"}},{"code_sha256_prefix":"a039ef466f295d02","entry":"conv_kernel_initializer","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/efficientnet_model.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/efficientnet_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a039ef466f295d02"}},{"code_sha256_prefix":"ea1b2be8595d0bb3","entry":"cutout","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/autoaugment.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/autoaugment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea1b2be8595d0bb3"}},{"code_sha256_prefix":"ec6986123a6a4894","entry":"dense_kernel_initializer","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/efficientnet_model.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/efficientnet_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec6986123a6a4894"}},{"code_sha256_prefix":"b54e5d5d7eccda19","entry":"distorted_bounding_box_crop","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/preprocessing.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b54e5d5d7eccda19"}},{"code_sha256_prefix":"d63756c8fcf336f4","entry":"drop_connect","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/utils.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d63756c8fcf336f4"}},{"code_sha256_prefix":"a7b35049ce7d1511","entry":"efficientnet_params","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/efficientnet_builder.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/efficientnet_builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7b35049ce7d1511"}},{"code_sha256_prefix":"6d17319dde175b8a","entry":"preprocess_for_eval","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/preprocessing.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d17319dde175b8a"}},{"code_sha256_prefix":"a923706b61b129ea","entry":"preprocess_for_train","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/preprocessing.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a923706b61b129ea"}},{"code_sha256_prefix":"9f7fd2caf2b48718","entry":"round_filters","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/efficientnet_model.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/efficientnet_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f7fd2caf2b48718"}},{"code_sha256_prefix":"918db160732a16d0","entry":"swish","repo":"cihangxie/SmoothAdversarialTraining","repo_kind":"official","path":"EfficientNet/efficientnet_builder.py","file_url":"https://github.com/cihangxie/SmoothAdversarialTraining/blob/HEAD/EfficientNet/efficientnet_builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"918db160732a16d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}