{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/metashift-a-dataset-of-datasets-for-1","title":"MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts","arxiv_id":"2202.06523","date":"2022-02-14","proceeding":"ICLR 2022 4","authors":["Weixin Liang","James Zou"],"abstract":"Understanding the performance of machine learning models across diverse data distributions is critically important for reliable applications. Motivated by this, there is a growing focus on curating benchmark datasets that capture distribution shifts. While valuable, the existing benchmarks are limited in that many of them only contain a small number of shifts and they lack systematic annotation about what is different across different shifts. We present MetaShift--a collection of 12,868 sets of natural images across 410 classes--to address this challenge. We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift. The key construction idea is to cluster images using its metadata, which provides context for each image (e.g. \"cats with cars\" or \"cats in bathroom\") that represent distinct data distributions. MetaShift has two important benefits: first, it contains orders of magnitude more natural data shifts than previously available. Second, it provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any two of its data sets. We demonstrate the utility of MetaShift in benchmarking several recent proposals for training models to be robust to data shifts. We find that the simple empirical risk minimization performs the best when shifts are moderate and no method had a systematic advantage for large shifts. We also show how MetaShift can help to visualize conflicts between data subsets during model training.","url_abs":"https://arxiv.org/abs/2202.06523v1","url_pdf":"https://arxiv.org/pdf/2202.06523v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"metashift-a-dataset-of-datasets-for-1","repo_url":"https://github.com/weixin-liang/metashift","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[{"slug":"metashift","name":"MetaShift","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2202.06523","atlas_url":"https://app.syntology.ai/?focus=2202.06523","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.06523"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/weixin-liang/metashift","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"a5e37b0cab1cbb28","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a5e37b0cab1cbb28"}},{"code_sha256_prefix":"3e901c4ed59c9f3a","entry":"build_subset_graph","repo":"weixin-liang/metashift","repo_kind":"official","path":"dataset/generate_full_MetaShift.py","file_url":"https://github.com/weixin-liang/metashift/blob/HEAD/dataset/generate_full_MetaShift.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e901c4ed59c9f3a"}},{"code_sha256_prefix":"27de4aa52fde48b8","entry":"node_str_to_subject_tag","repo":"weixin-liang/metashift","repo_kind":"official","path":"experiments/distribution_shift/main_generalization.py","file_url":"https://github.com/weixin-liang/metashift/blob/HEAD/experiments/distribution_shift/main_generalization.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27de4aa52fde48b8"}},{"code_sha256_prefix":"4bf7300ac8e005e7","entry":"parse_node_str","repo":"weixin-liang/metashift","repo_kind":"official","path":"dataset/generate_full_MetaShift.py","file_url":"https://github.com/weixin-liang/metashift/blob/HEAD/dataset/generate_full_MetaShift.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bf7300ac8e005e7"}},{"code_sha256_prefix":"ac8f33705a95e2de","entry":"draw_subject_set_graph","repo":"weixin-liang/metashift","repo_kind":"official","path":"dataset/generate_full_MetaShift.py","file_url":"https://github.com/weixin-liang/metashift/blob/HEAD/dataset/generate_full_MetaShift.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac8f33705a95e2de"}},{"code_sha256_prefix":"b4966e51c262718f","entry":"validate","repo":"weixin-liang/metashift","repo_kind":"official","path":"experiments/distribution_shift/main_generalization.py","file_url":"https://github.com/weixin-liang/metashift/blob/HEAD/experiments/distribution_shift/main_generalization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4966e51c262718f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}