{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distributionally-robust-neural-networks-for","title":"Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization","arxiv_id":"1911.08731","date":"2019-11-20","proceeding":null,"authors":["Shiori Sagawa","Pang Wei Koh","Tatsunori B. Hashimoto","Percy Liang"],"abstract":"Overparameterized neural networks can be highly accurate on average on an i.i.d. test set yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such groups). Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups. However, we find that naively applying group DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss also already has vanishing worst-case training loss. Instead, the poor worst-case performance arises from poor generalization on some groups. By coupling group DRO models with increased regularization---a stronger-than-typical L2 penalty or early stopping---we achieve substantially higher worst-group accuracies, with 10-40 percentage point improvements on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is important for worst-group generalization in the overparameterized regime, even if it is not needed for average generalization. Finally, we introduce a stochastic optimization algorithm, with convergence guarantees, to efficiently train group DRO models.","url_abs":"https://arxiv.org/abs/1911.08731v2","url_pdf":"https://arxiv.org/pdf/1911.08731v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://github.com/kohpangwei/group_DRO","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://worksheets.codalab.org/worksheets/0x621811fe446b49bb818293bae2ef88c0","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://github.com/facebookresearch/DomainBed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://github.com/haoxiang-wang/isr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://github.com/orparask/VS-Loss","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://github.com/ssagawa/overparam_spur_corr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://github.com/yangarbiter/dp-dg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"distributionally-robust-neural-networks-for","repo_url":"https://github.com/ys-zong/medfair","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"},{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/domain-generalization-on-nico-animal","task":"Domain Generalization","dataset":"NICO Animal","model":"DRO (Resnet-18)","rank_in_archive_order":5,"of":5,"metrics":{"Accuracy":"77.61"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-nico-vehicle","task":"Domain Generalization","dataset":"NICO Vehicle","model":"DRO (Resnet-18)","rank_in_archive_order":3,"of":5,"metrics":{"Accuracy":"77.61"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-pacs-2","task":"Domain Generalization","dataset":"PACS","model":"GroupDRO (Resnet-50, DomainBed)","rank_in_archive_order":61,"of":133,"metrics":{"Average Accuracy":"84.4"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1911.08731","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.08731"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kohpangwei/group_DRO","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/DomainBed","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangarbiter/dp-dg","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ssagawa/overparam_spur_corr","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ys-zong/medfair","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haoxiang-wang/isr","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/orparask/VS-Loss","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://worksheets.codalab.org/worksheets/0x621811fe446b49bb818293bae2ef88c0","reach":null}],"summary":{"ran":3,"ran_draft_wrong":4,"ran_honours":2,"unverified":1},"by_repo_kind":{"listed":{"samples":10,"ran":9,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"ee24fe60820363aa","entry":"ISRClassifier","repo":"haoxiang-wang/isr","repo_kind":"listed","path":"real_datasets/isr.py","file_url":"https://github.com/haoxiang-wang/isr/blob/HEAD/real_datasets/isr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee24fe60820363aa"}},{"code_sha256_prefix":"c42a893093b6fd69","entry":"LossComputer","repo":"ssagawa/overparam_spur_corr","repo_kind":"listed","path":"loss.py","file_url":"https://github.com/ssagawa/overparam_spur_corr/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c42a893093b6fd69"}},{"code_sha256_prefix":"d1ceae741f009c45","entry":"LossComputer","repo":"orparask/VS-Loss","repo_kind":"listed","path":"group_imbalance/loss.py","file_url":"https://github.com/orparask/VS-Loss/blob/HEAD/group_imbalance/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1ceae741f009c45"}},{"code_sha256_prefix":"e4aa557c0470b486","entry":"check_clf","repo":"haoxiang-wang/isr","repo_kind":"listed","path":"real_datasets/isr.py","file_url":"https://github.com/haoxiang-wang/isr/blob/HEAD/real_datasets/isr.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4aa557c0470b486"}},{"code_sha256_prefix":"c66e837fdb8da4e1","entry":"check_labels","repo":"haoxiang-wang/isr","repo_kind":"listed","path":"real_datasets/isr.py","file_url":"https://github.com/haoxiang-wang/isr/blob/HEAD/real_datasets/isr.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c66e837fdb8da4e1"}},{"code_sha256_prefix":"953fdd3b2318ba41","entry":"estimate_covs","repo":"haoxiang-wang/isr","repo_kind":"listed","path":"real_datasets/isr.py","file_url":"https://github.com/haoxiang-wang/isr/blob/HEAD/real_datasets/isr.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"953fdd3b2318ba41"}},{"code_sha256_prefix":"a26fa43681625cbf","entry":"estimate_means","repo":"haoxiang-wang/isr","repo_kind":"listed","path":"real_datasets/isr.py","file_url":"https://github.com/haoxiang-wang/isr/blob/HEAD/real_datasets/isr.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a26fa43681625cbf"}},{"code_sha256_prefix":"30fd09a08819dcf5","entry":"feature_transform","repo":"haoxiang-wang/isr","repo_kind":"listed","path":"real_datasets/isr.py","file_url":"https://github.com/haoxiang-wang/isr/blob/HEAD/real_datasets/isr.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30fd09a08819dcf5"}},{"code_sha256_prefix":"78b0af73923a5d41","entry":"to_numpy","repo":"haoxiang-wang/isr","repo_kind":"listed","path":"real_datasets/isr.py","file_url":"https://github.com/haoxiang-wang/isr/blob/HEAD/real_datasets/isr.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78b0af73923a5d41"}},{"code_sha256_prefix":"e944e38b6f666c5d","entry":"GroupDRO","repo":"facebookresearch/DomainBed","repo_kind":"listed","path":"domainbed/algorithms.py","file_url":"https://github.com/facebookresearch/DomainBed/blob/HEAD/domainbed/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e944e38b6f666c5d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}