{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-suite-data-centric-identification-of-in","title":"Data-SUITE: Data-centric identification of in-distribution incongruous examples","arxiv_id":"2202.08836","date":"2022-02-17","proceeding":null,"authors":["Nabeel Seedat","Jonathan Crabbé","Mihaela van der Schaar"],"abstract":"Systematic quantification of data quality is critical for consistent model performance. Prior works have focused on out-of-distribution data. Instead, we tackle an understudied yet equally important problem of characterizing incongruous regions of in-distribution (ID) data, which may arise from feature space heterogeneity. To this end, we propose a paradigm shift with Data-SUITE: a data-centric AI framework to identify these regions, independent of a task-specific model. Data-SUITE leverages copula modeling, representation learning, and conformal prediction to build feature-wise confidence interval estimators based on a set of training instances. These estimators can be used to evaluate the congruence of test instances with respect to the training set, to answer two practically useful questions: (1) which test instances will be reliably predicted by a model trained with the training instances? and (2) can we identify incongruous regions of the feature space so that data owners understand the data's limitations or guide future data collection? We empirically validate Data-SUITE's performance and coverage guarantees and demonstrate on cross-site medical data, biased data, and data with concept drift, that Data-SUITE best identifies ID regions where a downstream model may be reliable (independent of said model). We also illustrate how these identified regions can provide insights into datasets and highlight their limitations.","url_abs":"https://arxiv.org/abs/2202.08836v3","url_pdf":"https://arxiv.org/pdf/2202.08836v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-suite-data-centric-identification-of-in","repo_url":"https://github.com/seedatnabeel/data-suite","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"conformal-prediction","task_name":"Conformal Prediction"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.08836","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.08836"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seedatnabeel/data-suite","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f7ce604d7f4dcace","entry":"all_equal2","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/models/mcd.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/models/mcd.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7ce604d7f4dcace"}},{"code_sha256_prefix":"7357cd13309d3f31","entry":"compute_representation","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/models/representation.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/models/representation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7357cd13309d3f31"}},{"code_sha256_prefix":"9d9f5093936c406a","entry":"covariance_comparison","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/utils/data_utils.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d9f5093936c406a"}},{"code_sha256_prefix":"34f7beadd5fda2a1","entry":"get_mse_table","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/utils/graphics.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/utils/graphics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34f7beadd5fda2a1"}},{"code_sha256_prefix":"3f0f7bcbd34e50f4","entry":"inlier_outlier_dicts","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/utils/helpers.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/utils/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f0f7bcbd34e50f4"}},{"code_sha256_prefix":"8515fb9ae1ae6e7b","entry":"representation_class_based","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/models/representation.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/models/representation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8515fb9ae1ae6e7b"}},{"code_sha256_prefix":"8827b14a2d884cb0","entry":"return_diagonal","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/utils/data_utils.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8827b14a2d884cb0"}},{"code_sha256_prefix":"ab6621513840038b","entry":"sample_copula","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/models/benchmarks.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/models/benchmarks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab6621513840038b"}},{"code_sha256_prefix":"db182a4e5b31bf62","entry":"scaler","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/utils/data_utils.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db182a4e5b31bf62"}},{"code_sha256_prefix":"58c9f9694a3b2975","entry":"sort_cis_all","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/utils/helpers.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/utils/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58c9f9694a3b2975"}},{"code_sha256_prefix":"53035afb7a99185e","entry":"sort_cis_synth","repo":"seedatnabeel/data-suite","repo_kind":"official","path":"src/utils/helpers.py","file_url":"https://github.com/seedatnabeel/data-suite/blob/HEAD/src/utils/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53035afb7a99185e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}