{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-debugging-with-shapley-importance-over","title":"Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelines","arxiv_id":"2204.11131","date":"2022-04-23","proceeding":null,"authors":["Bojan Karlaš","David Dao","Matteo Interlandi","Bo Li","Sebastian Schelter","Wentao Wu","Ce Zhang"],"abstract":"Developing modern machine learning (ML) applications is data-centric, of which one fundamental challenge is to understand the influence of data quality to ML training -- \"Which training examples are 'guilty' in making the trained ML model predictions inaccurate or unfair?\" Modeling data influence for ML training has attracted intensive interest over the last decade, and one popular framework is to compute the Shapley value of each training example with respect to utilities such as validation accuracy and fairness of the trained ML model. Unfortunately, despite recent intensive interest and research, existing methods only consider a single ML model \"in isolation\" and do not consider an end-to-end ML pipeline that consists of data transformations, feature extractors, and ML training. We present DataScope (ease.ml/datascope), the first system that efficiently computes Shapley values of training examples over an end-to-end ML pipeline, and illustrate its applications in data debugging for ML training. To this end, we first develop a novel algorithmic framework that computes Shapley value over a specific family of ML pipelines that we call canonical pipelines: a positive relational algebra query followed by a K-nearest-neighbor (KNN) classifier. We show that, for many subfamilies of canonical pipelines, computing Shapley value is in PTIME, contrasting the exponential complexity of computing Shapley value in general. We then put this to practice -- given an sklearn pipeline, we approximate it with a canonical pipeline to use as a proxy. We conduct extensive experiments illustrating different use cases and utilities. Our results show that DataScope is up to four orders of magnitude faster over state-of-the-art Monte Carlo-based methods, while being comparably, and often even more, effective in data debugging.","url_abs":"https://arxiv.org/abs/2204.11131v2","url_pdf":"https://arxiv.org/pdf/2204.11131v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-debugging-with-shapley-importance-over","repo_url":"https://github.com/easeml/datascope","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"fairness","task_name":"Fairness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2204.11131","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.11131"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/easeml/datascope","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c391b365af831681","entry":"checknan","repo":"easeml/datascope","repo_kind":"official","path":"datascope/importance/shapley.py","file_url":"https://github.com/easeml/datascope/blob/HEAD/datascope/importance/shapley.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c391b365af831681"}},{"code_sha256_prefix":"33dac2316da6e2d5","entry":"factorize_provenance","repo":"easeml/datascope","repo_kind":"official","path":"datascope/importance/shapley.py","file_url":"https://github.com/easeml/datascope/blob/HEAD/datascope/importance/shapley.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33dac2316da6e2d5"}},{"code_sha256_prefix":"8f68bb355d9a41e0","entry":"get_test_batch_size","repo":"easeml/datascope","repo_kind":"official","path":"datascope/importance/shapley.py","file_url":"https://github.com/easeml/datascope/blob/HEAD/datascope/importance/shapley.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f68bb355d9a41e0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}