{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-data-addition-dilemma","title":"The Data Addition Dilemma","arxiv_id":"2408.04154","date":"2024-08-08","proceeding":null,"authors":["Judy Hanwen Shen","Inioluwa Deborah Raji","Irene Y. Chen"],"abstract":"In many machine learning for healthcare tasks, standard datasets are constructed by amassing data across many, often fundamentally dissimilar, sources. But when does adding more data help, and when does it hinder progress on desired model outcomes in real-world settings? We identify this situation as the \\textit{Data Addition Dilemma}, demonstrating that adding training data in this multi-source scaling context can at times result in reduced overall accuracy, uncertain fairness outcomes, and reduced worst-subgroup performance. We find that this possibly arises from an empirically observed trade-off between model performance improvements due to data scaling and model deterioration from distribution shift. We thus establish baseline strategies for navigating this dilemma, introducing distribution shift heuristics to guide decision-making on which data sources to add in data scaling, in order to yield the expected model performance improvements. We conclude with a discussion of the required considerations for data collection and suggestions for studying data composition and scale in the age of increasingly larger models.","url_abs":"https://arxiv.org/abs/2408.04154v1","url_pdf":"https://arxiv.org/pdf/2408.04154v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-data-addition-dilemma","repo_url":"https://github.com/the-chen-lab/data-addition-dilemma","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"fairness","task_name":"Fairness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.04154","atlas_url":"https://app.syntology.ai/?focus=2408.04154","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.04154"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/the-chen-lab/data-addition-dilemma","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"official":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"8e277d9a0db29745","entry":"flatten","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_dip.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_dip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e277d9a0db29745"}},{"code_sha256_prefix":"9efba4f9526a4af4","entry":"get_Xy","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_mixture.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9efba4f9526a4af4"}},{"code_sha256_prefix":"5893354afbcc35a6","entry":"get_clean_race","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_dip.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_dip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5893354afbcc35a6"}},{"code_sha256_prefix":"4ff08a649412b970","entry":"get_data_for_state","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"folktables_exp/run_scaling.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/folktables_exp/run_scaling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ff08a649412b970"}},{"code_sha256_prefix":"55187e06900bef17","entry":"get_group","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_dip.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_dip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55187e06900bef17"}},{"code_sha256_prefix":"cf9d5d8d97e9a8ff","entry":"get_group","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_sequential.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_sequential.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf9d5d8d97e9a8ff"}},{"code_sha256_prefix":"3d4a891116d14206","entry":"get_hospital","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"kl_utils.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/kl_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d4a891116d14206"}},{"code_sha256_prefix":"2235175d7038580c","entry":"group_accuracy","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"folktables_exp/metrics.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/folktables_exp/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2235175d7038580c"}},{"code_sha256_prefix":"b8d7fff533afdcd3","entry":"group_accuracy_ot","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"folktables_exp/metrics.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/folktables_exp/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8d7fff533afdcd3"}},{"code_sha256_prefix":"fa4bb6e5d361ce50","entry":"model_choice","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"folktables_exp/metrics.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/folktables_exp/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa4bb6e5d361ce50"}},{"code_sha256_prefix":"92f8ecbd9aea7e44","entry":"partition_array","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"kl_utils.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/kl_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92f8ecbd9aea7e44"}},{"code_sha256_prefix":"bb6dd2f3e62967b9","entry":"run_experiment","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_mixture.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb6dd2f3e62967b9"}},{"code_sha256_prefix":"b3ee0f411e006d36","entry":"run_model","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_mixture.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3ee0f411e006d36"}},{"code_sha256_prefix":"797bb0509e425320","entry":"run_yelp_exp_prep","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_sequential.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_sequential.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"797bb0509e425320"}},{"code_sha256_prefix":"cfe5cb3002787234","entry":"yelp_seq_data_prep","repo":"the-chen-lab/data-addition-dilemma","repo_kind":"official","path":"Yelp-MIMIC/run_sequential.py","file_url":"https://github.com/the-chen-lab/data-addition-dilemma/blob/HEAD/Yelp-MIMIC/run_sequential.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfe5cb3002787234"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}