{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-dataset-multiplicity-problem-how","title":"The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions","arxiv_id":"2304.10655","date":"2023-04-20","proceeding":null,"authors":["Anna P. Meyer","Aws Albarghouthi","Loris D'Antoni"],"abstract":"We introduce dataset multiplicity, a way to study how inaccuracies, uncertainty, and social bias in training datasets impact test-time predictions. The dataset multiplicity framework asks a counterfactual question of what the set of resultant models (and associated test-time predictions) would be if we could somehow access all hypothetical, unbiased versions of the dataset. We discuss how to use this framework to encapsulate various sources of uncertainty in datasets' factualness, including systemic social bias, data collection practices, and noisy labels or features. We show how to exactly analyze the impacts of dataset multiplicity for a specific model architecture and type of uncertainty: linear models with label errors. Our empirical analysis shows that real-world datasets, under reasonable assumptions, contain many test samples whose predictions are affected by dataset multiplicity. Furthermore, the choice of domain-specific dataset multiplicity definition determines what samples are affected, and whether different demographic groups are disparately impacted. Finally, we discuss implications of dataset multiplicity for machine learning practice and research, including considerations for when model outcomes should not be trusted.","url_abs":"https://arxiv.org/abs/2304.10655v1","url_pdf":"https://arxiv.org/pdf/2304.10655v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-dataset-multiplicity-problem-how","repo_url":"https://github.com/annapmeyer/linear-bias-certification","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"counterfactual"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.10655","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.10655"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/annapmeyer/linear-bias-certification","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"727a4d40da335ca6","entry":"compute_matr_prod","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"linear_eq.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/linear_eq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"727a4d40da335ca6"}},{"code_sha256_prefix":"012a306c56b31562","entry":"int_add","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"interval.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/interval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"012a306c56b31562"}},{"code_sha256_prefix":"1efe7ec64a1b1d12","entry":"int_mult","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"interval.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/interval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1efe7ec64a1b1d12"}},{"code_sha256_prefix":"315351f17b28c6f3","entry":"int_subtr","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"interval.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/interval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"315351f17b28c6f3"}},{"code_sha256_prefix":"b24db22ccc376215","entry":"mnist_special","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b24db22ccc376215"}},{"code_sha256_prefix":"667ed4dc6b3c5560","entry":"out_of_box","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"linear.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/linear.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"667ed4dc6b3c5560"}},{"code_sha256_prefix":"25ec78baefc8e9af","entry":"probe_accuracy","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"linear_eq.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/linear_eq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25ec78baefc8e9af"}},{"code_sha256_prefix":"e2bf6961de2bcff0","entry":"probe_accuracy_cont","repo":"annapmeyer/linear-bias-certification","repo_kind":"official","path":"linear_eq.py","file_url":"https://github.com/annapmeyer/linear-bias-certification/blob/HEAD/linear_eq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2bf6961de2bcff0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}