{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-many-faces-of-robustness-a-critical","title":"The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization","arxiv_id":"2006.16241","date":"2020-06-29","proceeding":"ICCV 2021 10","authors":["Dan Hendrycks","Steven Basart","Norman Mu","Saurav Kadavath","Frank Wang","Evan Dorundo","Rahul Desai","Tyler Zhu","Samyak Parajuli","Mike Guo","Dawn Song","Jacob Steinhardt","Justin Gilmer"],"abstract":"We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000 times more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.","url_abs":"https://arxiv.org/abs/2006.16241v3","url_pdf":"https://arxiv.org/pdf/2006.16241v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-many-faces-of-robustness-a-critical","repo_url":"https://github.com/hendrycks/imagenet-r","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"}],"methods":[],"datasets_introduced":[{"slug":"imagenet-r","name":"ImageNet-R","full_name":"ImageNet-Rendition"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/domain-generalization-on-imagenet-c","task":"Domain Generalization","dataset":"ImageNet-C","model":"DeepAugment (ResNet-50)","rank_in_archive_order":36,"of":47,"metrics":{"mean Corruption Error (mCE)":" 60.4"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-imagenet-r","task":"Domain Generalization","dataset":"ImageNet-R","model":"DeepAugment+AugMix (ResNet-50)","rank_in_archive_order":29,"of":39,"metrics":{"Top-1 Error Rate":" 53.2 "},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-imagenet-r","task":"Domain Generalization","dataset":"ImageNet-R","model":"DeepAugment (ResNet-50)","rank_in_archive_order":33,"of":39,"metrics":{"Top-1 Error Rate":"57.8"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-vizwiz","task":"Domain Generalization","dataset":"VizWiz-Classification","model":"ResNet-50 (deepaugment)","rank_in_archive_order":30,"of":90,"metrics":{"Accuracy - All Images":"41.3","Accuracy - Clean Images":"46","Accuracy - Corrupted Images":"34.9"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-vizwiz","task":"Domain Generalization","dataset":"VizWiz-Classification","model":"ResNet-50 (deepaugment+augmix)","rank_in_archive_order":35,"of":90,"metrics":{"Accuracy - All Images":"40.3","Accuracy - Clean Images":"44.5","Accuracy - Corrupted Images":"34.1"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2006.16241","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.16241"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hendrycks/imagenet-r","reach":null}],"summary":{"ran":1,"ran_draft_wrong":12,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"90f0b6606b7bec55","entry":"AugMix","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90f0b6606b7bec55"}},{"code_sha256_prefix":"48502588e268dd8f","entry":"autocontrast","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48502588e268dd8f"}},{"code_sha256_prefix":"7fa83d532dab3fa4","entry":"brightness","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fa83d532dab3fa4"}},{"code_sha256_prefix":"f53b32f9516fdaa1","entry":"color","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f53b32f9516fdaa1"}},{"code_sha256_prefix":"6e50ec13656740c3","entry":"contrast","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e50ec13656740c3"}},{"code_sha256_prefix":"cbb8d1e8526fa7eb","entry":"equalize","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbb8d1e8526fa7eb"}},{"code_sha256_prefix":"e51e7aefc03823aa","entry":"posterize","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e51e7aefc03823aa"}},{"code_sha256_prefix":"f3de03c4a896f7e5","entry":"rotate","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3de03c4a896f7e5"}},{"code_sha256_prefix":"c55e7427481ee18e","entry":"sharpness","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c55e7427481ee18e"}},{"code_sha256_prefix":"8b301c76d5b96b5f","entry":"shear_x","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b301c76d5b96b5f"}},{"code_sha256_prefix":"74de255c18b450ef","entry":"shear_y","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74de255c18b450ef"}},{"code_sha256_prefix":"ea3db097ab0c59d3","entry":"solarize","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea3db097ab0c59d3"}},{"code_sha256_prefix":"68ee08ee193d291c","entry":"translate_x","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68ee08ee193d291c"}},{"code_sha256_prefix":"49792e8ad13a4ec5","entry":"translate_y","repo":"hendrycks/imagenet-r","repo_kind":"official","path":"deepfashion_remix/augmix.py","file_url":"https://github.com/hendrycks/imagenet-r/blob/HEAD/deepfashion_remix/augmix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49792e8ad13a4ec5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}