{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/wilds-a-benchmark-of-in-the-wild-distribution","title":"WILDS: A Benchmark of in-the-Wild Distribution Shifts","arxiv_id":"2012.07421","date":"2020-12-14","proceeding":null,"authors":["Pang Wei Koh","Shiori Sagawa","Henrik Marklund","Sang Michael Xie","Marvin Zhang","Akshay Balsubramani","Weihua Hu","Michihiro Yasunaga","Richard Lanas Phillips","Irena Gao","Tony Lee","Etienne David","Ian Stavness","Wei Guo","Berton A. Earnshaw","Imran S. Haque","Sara Beery","Jure Leskovec","Anshul Kundaje","Emma Pierson","Sergey Levine","Chelsea Finn","Percy Liang"],"abstract":"Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchmark of 10 datasets reflecting a diverse range of distribution shifts that naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at https://wilds.stanford.edu.","url_abs":"https://arxiv.org/abs/2012.07421v3","url_pdf":"https://arxiv.org/pdf/2012.07421v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"wilds-a-benchmark-of-in-the-wild-distribution","repo_url":"https://github.com/p-lambda/wilds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"wilds-a-benchmark-of-in-the-wild-distribution","repo_url":"https://github.com/facebookresearch/DomainBed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"wilds-a-benchmark-of-in-the-wild-distribution","repo_url":"https://github.com/hlzhang109/ddg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"wilds-a-benchmark-of-in-the-wild-distribution","repo_url":"https://github.com/qiaoruiyt/noiserobustdg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"wilds-a-benchmark-of-in-the-wild-distribution","repo_url":"https://github.com/skyve2012/DBA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"wilds-a-benchmark-of-in-the-wild-distribution","repo_url":"https://github.com/tigrangalstyan/wilds","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[],"datasets_introduced":[{"slug":"wilds","name":"Wilds","full_name":""},{"slug":"iwildcam2020-wilds","name":"iWildCam2020-WILDS","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-iwildcam2020-wilds","task":"Image Classification","dataset":"iWildCam2020-WILDS","model":"Empirical Risk Minimization (ERM)","rank_in_archive_order":5,"of":6,"metrics":{"Accuracy (Top-1)":"71.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2012.07421","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2012.07421"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/DomainBed","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/skyve2012/DBA","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tigrangalstyan/wilds","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/p-lambda/wilds","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hlzhang109/ddg","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qiaoruiyt/noiserobustdg","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e32762f7bf5137a0","entry":"get_eval_loader","repo":"p-lambda/wilds","repo_kind":"official","path":"wilds/common/data_loaders.py","file_url":"https://github.com/p-lambda/wilds/blob/HEAD/wilds/common/data_loaders.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e32762f7bf5137a0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}