{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dataset-cartography-mapping-and-diagnosing","title":"Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics","arxiv_id":"2009.10795","date":"2020-09-22","proceeding":"EMNLP 2020 11","authors":["Swabha Swayamdipta","Roy Schwartz","Nicholas Lourie","Yizhong Wang","Hannaneh Hajishirzi","Noah A. Smith","Yejin Choi"],"abstract":"Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Maps---a model-based tool to characterize and diagnose datasets. We leverage a largely ignored source of information: the behavior of the model on individual instances during training (training dynamics) for building data maps. This yields two intuitive measures for each example---the model's confidence in the true class, and the variability of this confidence across epochs---obtained in a single run of training. Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics. First, our data maps show the presence of \"ambiguous\" regions with respect to the model, which contribute the most towards out-of-distribution generalization. Second, the most populous regions in the data are \"easy to learn\" for the model, and play an important role in model optimization. Finally, data maps uncover a region with instances that the model finds \"hard to learn\"; these often correspond to labeling errors. Our results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization.","url_abs":"https://arxiv.org/abs/2009.10795v2","url_pdf":"https://arxiv.org/pdf/2009.10795v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dataset-cartography-mapping-and-diagnosing","repo_url":"https://github.com/allenai/cartography","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dataset-cartography-mapping-and-diagnosing","repo_url":"https://github.com/DorBernsohn/TrainingDynamics","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"dataset-cartography-mapping-and-diagnosing","repo_url":"https://github.com/dyahadila/ood_cartography","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"dataset-cartography-mapping-and-diagnosing","repo_url":"https://github.com/szhang42/uneven_training_data","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"dataset-cartography-mapping-and-diagnosing","repo_url":"https://github.com/beyondguo/TrainingDynamics","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"dataset-cartography-mapping-and-diagnosing","repo_url":"https://github.com/eliorc/tavolo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"model-optimization","task_name":"Model Optimization"},{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2009.10795","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.10795"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/cartography","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/szhang42/uneven_training_data","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DorBernsohn/TrainingDynamics","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/beyondguo/TrainingDynamics","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eliorc/tavolo","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dyahadila/ood_cartography","reach":null}],"summary":{"ran":1,"ran_violates":2,"ran_fixture":2,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1},"listed":{"samples":8,"ran":7,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"63892da82b9ce195","entry":"DataMapCallback","repo":"eliorc/tavolo","repo_kind":"listed","path":"tavolo/learning.py","file_url":"https://github.com/eliorc/tavolo/blob/HEAD/tavolo/learning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63892da82b9ce195"}},{"code_sha256_prefix":"b53e0e5cc0667b0b","entry":"compute_correctness","repo":"szhang42/uneven_training_data","repo_kind":"listed","path":"cartography/selection/train_dy_filtering.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/cartography/selection/train_dy_filtering.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b53e0e5cc0667b0b"}},{"code_sha256_prefix":"ebca92d1e4b67794","entry":"compute_correctness","repo":"beyondguo/TrainingDynamics","repo_kind":"listed","path":"dy_filtering.py","file_url":"https://github.com/beyondguo/TrainingDynamics/blob/HEAD/dy_filtering.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebca92d1e4b67794"}},{"code_sha256_prefix":"bd88c7f0a36c7e6a","entry":"compute_forgetfulness","repo":"szhang42/uneven_training_data","repo_kind":"listed","path":"cartography/selection/train_dy_filtering.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/cartography/selection/train_dy_filtering.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd88c7f0a36c7e6a"}},{"code_sha256_prefix":"270d424da751f2cf","entry":"compute_forgetfulness","repo":"beyondguo/TrainingDynamics","repo_kind":"listed","path":"dy_filtering.py","file_url":"https://github.com/beyondguo/TrainingDynamics/blob/HEAD/dy_filtering.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"270d424da751f2cf"}},{"code_sha256_prefix":"9aac3b7014417968","entry":"compute_train_dy_metrics","repo":"szhang42/uneven_training_data","repo_kind":"listed","path":"cartography/selection/train_dy_filtering.py","file_url":"https://github.com/szhang42/uneven_training_data/blob/HEAD/cartography/selection/train_dy_filtering.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9aac3b7014417968"}},{"code_sha256_prefix":"fbb1ac1ee7b9badd","entry":"compute_train_dy_metrics","repo":"beyondguo/TrainingDynamics","repo_kind":"listed","path":"dy_filtering.py","file_url":"https://github.com/beyondguo/TrainingDynamics/blob/HEAD/dy_filtering.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbb1ac1ee7b9badd"}},{"code_sha256_prefix":"c1dc8ee8436bbb66","entry":"adapted_glue_compute_metrics","repo":"allenai/cartography","repo_kind":"official","path":"cartography/classification/glue_utils.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/classification/glue_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1dc8ee8436bbb66"}},{"code_sha256_prefix":"13e0257678f83c36","entry":"compute_train_dy_metrics_per_epoch","repo":"dyahadila/ood_cartography","repo_kind":"listed","path":"cartography/selection/train_dy_filtering.py","file_url":"https://github.com/dyahadila/ood_cartography/blob/HEAD/cartography/selection/train_dy_filtering.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13e0257678f83c36"}},{"code_sha256_prefix":"0356041f7d4f1629","entry":"convert_mc_examples_to_features","repo":"allenai/cartography","repo_kind":"official","path":"cartography/classification/multiple_choice_utils.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/classification/multiple_choice_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0356041f7d4f1629"}},{"code_sha256_prefix":"18829c005e571204","entry":"convert_string_to_unique_number","repo":"allenai/cartography","repo_kind":"official","path":"cartography/data_utils_glue.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/data_utils_glue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"18829c005e571204"}},{"code_sha256_prefix":"d976a22c80de6352","entry":"convert_tsv_entries_to_dataframe","repo":"allenai/cartography","repo_kind":"official","path":"cartography/data_utils.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d976a22c80de6352"}},{"code_sha256_prefix":"8668f4254aab1786","entry":"determine_categories_by_fields","repo":"allenai/cartography","repo_kind":"official","path":"cartography/classification/diagnostics_evaluation.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/classification/diagnostics_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8668f4254aab1786"}},{"code_sha256_prefix":"17a0e1fc45469b30","entry":"label_balance","repo":"allenai/cartography","repo_kind":"official","path":"cartography/classification/diagnostics_evaluation.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/classification/diagnostics_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17a0e1fc45469b30"}},{"code_sha256_prefix":"b03cdf908937ca31","entry":"read_glue_tsv","repo":"allenai/cartography","repo_kind":"official","path":"cartography/data_utils_glue.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/data_utils_glue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b03cdf908937ca31"}},{"code_sha256_prefix":"d4036398be7d8fc9","entry":"read_jsonl","repo":"allenai/cartography","repo_kind":"official","path":"cartography/data_utils.py","file_url":"https://github.com/allenai/cartography/blob/HEAD/cartography/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d4036398be7d8fc9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}