{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/why-did-the-model-fail-attributing-model","title":"\"Why did the Model Fail?\": Attributing Model Performance Changes to Distribution Shifts","arxiv_id":"2210.10769","date":"2022-10-19","proceeding":null,"authors":["Haoran Zhang","Harvineet Singh","Marzyeh Ghassemi","Shalmali Joshi"],"abstract":"Machine learning models frequently experience performance drops under distribution shifts. The underlying cause of such shifts may be multiple simultaneous factors such as changes in data quality, differences in specific covariate distributions, or changes in the relationship between label and features. When a model does fail during deployment, attributing performance change to these factors is critical for the model developer to identify the root cause and take mitigating actions. In this work, we introduce the problem of attributing performance differences between environments to distribution shifts in the underlying data generating mechanisms. We formulate the problem as a cooperative game where the players are distributions. We define the value of a set of distributions to be the change in model performance when only this set of distributions has changed between environments, and derive an importance weighting method for computing the value of an arbitrary set of distributions. The contribution of each distribution to the total performance change is then quantified as its Shapley value. We demonstrate the correctness and utility of our method on synthetic, semi-synthetic, and real-world case studies, showing its effectiveness in attributing performance changes to a wide range of distribution shifts.","url_abs":"https://arxiv.org/abs/2210.10769v3","url_pdf":"https://arxiv.org/pdf/2210.10769v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"why-did-the-model-fail-attributing-model","repo_url":"https://github.com/mlforhealth/expl_perf_drop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.10769","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.10769"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/MLforHealth/expl_","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlforhealth/expl_perf_drop","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_honours":1,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5a9f7480b832c15a","entry":"combinations_base","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/experiments.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/experiments.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a9f7480b832c15a"}},{"code_sha256_prefix":"f2059a864ab0dc16","entry":"get_slurm_jobs","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/launchers.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/launchers.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2059a864ab0dc16"}},{"code_sha256_prefix":"ed5bdae1388ade09","entry":"AUROC","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/metrics.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed5bdae1388ade09"}},{"code_sha256_prefix":"4b746b056a598558","entry":"accuracy","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/metrics.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b746b056a598558"}},{"code_sha256_prefix":"bc6dc62bdfa1e0d9","entry":"block_until_running","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/launchers.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/launchers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc6dc62bdfa1e0d9"}},{"code_sha256_prefix":"f408cb23a303554c","entry":"brier","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/metrics.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f408cb23a303554c"}},{"code_sha256_prefix":"f64641d62dc45aa9","entry":"combinations","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/experiments.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/experiments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f64641d62dc45aa9"}},{"code_sha256_prefix":"73d0cb5bbc5c4f0c","entry":"get_hparams","repo":"mlforhealth/expl_perf_drop","repo_kind":"official","path":"expl_perf_drop/experiments.py","file_url":"https://github.com/mlforhealth/expl_perf_drop/blob/HEAD/expl_perf_drop/experiments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73d0cb5bbc5c4f0c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}