{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-iq-characterizing-subgroups-with","title":"Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data","arxiv_id":"2210.13043","date":"2022-10-24","proceeding":null,"authors":["Nabeel Seedat","Jonathan Crabbé","Ioana Bica","Mihaela van der Schaar"],"abstract":"High model performance, on average, can hide that models may systematically underperform on subgroups of the data. We consider the tabular setting, which surfaces the unique issue of outcome heterogeneity - this is prevalent in areas such as healthcare, where patients with similar features can have different outcomes, thus making reliable predictions challenging. To tackle this, we propose Data-IQ, a framework to systematically stratify examples into subgroups with respect to their outcomes. We do this by analyzing the behavior of individual examples during training, based on their predictive confidence and, importantly, the aleatoric (data) uncertainty. Capturing the aleatoric uncertainty permits a principled characterization and then subsequent stratification of data examples into three distinct subgroups (Easy, Ambiguous, Hard). We experimentally demonstrate the benefits of Data-IQ on four real-world medical datasets. We show that Data-IQ's characterization of examples is most robust to variation across similarly performant (yet different) models, compared to baselines. Since Data-IQ can be used with any ML model (including neural networks, gradient boosting etc.), this property ensures consistency of data characterization, while allowing flexible model selection. Taking this a step further, we demonstrate that the subgroups enable us to construct new approaches to both feature acquisition and dataset selection. Furthermore, we highlight how the subgroups can inform reliable model usage, noting the significant impact of the Ambiguous subgroup on model generalization.","url_abs":"https://arxiv.org/abs/2210.13043v1","url_pdf":"https://arxiv.org/pdf/2210.13043v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-iq-characterizing-subgroups-with","repo_url":"https://github.com/seedatnabeel/data-iq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"data-iq-characterizing-subgroups-with","repo_url":"https://github.com/vanderschaarlab/data-iq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.13043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.13043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vanderschaarlab/data-iq","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seedatnabeel/data-iq","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b882b7743c5154c","entry":"accuracy","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/models/cnn_models.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/models/cnn_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b882b7743c5154c"}},{"code_sha256_prefix":"74b9aada2ba5b3c6","entry":"evaluate_model","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/group_dro_helpers.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/group_dro_helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74b9aada2ba5b3c6"}},{"code_sha256_prefix":"4a9eeb01ebd75af0","entry":"get_data","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/exp3_helpers.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/exp3_helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a9eeb01ebd75af0"}},{"code_sha256_prefix":"baf5730360c8aca7","entry":"get_dataiq_scores","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/utils.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"baf5730360c8aca7"}},{"code_sha256_prefix":"43a03ef033b2e718","entry":"get_grad_scores","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/utils.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43a03ef033b2e718"}},{"code_sha256_prefix":"280c66ad0e1e3bad","entry":"get_scores_generic","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/utils.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"280c66ad0e1e3bad"}},{"code_sha256_prefix":"f9bc3c2b6ce7ff3d","entry":"load_adult_data","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"tutorial/demo_data.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/tutorial/demo_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9bc3c2b6ce7ff3d"}},{"code_sha256_prefix":"a7898e3f93e6f24f","entry":"load_dataset","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/data_loader.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/data_loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7898e3f93e6f24f"}},{"code_sha256_prefix":"997137bb63769364","entry":"load_seer_cutract_dataset","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/data_loader.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/data_loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"997137bb63769364"}},{"code_sha256_prefix":"155df0192be9e607","entry":"load_seer_cutract_dataset","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/exp3_helpers.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/exp3_helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"155df0192be9e607"}},{"code_sha256_prefix":"89bd42887c6ec15c","entry":"to_device","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/models/cnn_models.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/models/cnn_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89bd42887c6ec15c"}},{"code_sha256_prefix":"369603fff1a7ea66","entry":"train_loop","repo":"seedatnabeel/data-iq","repo_kind":"official","path":"src/utils/group_dro_helpers.py","file_url":"https://github.com/seedatnabeel/data-iq/blob/HEAD/src/utils/group_dro_helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"369603fff1a7ea66"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}