{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/counterfactually-augmented-snli-training-data","title":"Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data","arxiv_id":"2010.04762","date":"2020-10-09","proceeding":"EMNLP (insights) 2020 11","authors":["William Huang","Haokun Liu","Samuel R. Bowman"],"abstract":"A growing body of work shows that models exploit annotation artifacts to achieve state-of-the-art performance on standard crowdsourced benchmarks---datasets collected from crowdworkers to create an evaluation task---while still failing on out-of-domain examples for the same task. Recent work has explored the use of counterfactually-augmented data---data built by minimally editing a set of seed examples to yield counterfactual labels---to augment training data associated with these benchmarks and build more robust classifiers that generalize better. However, Khashabi et al. (2020) find that this type of augmentation yields little benefit on reading comprehension tasks when controlling for dataset size and cost of collection. We build upon this work by using English natural language inference data to test model generalization and robustness and find that models trained on a counterfactually-augmented SNLI dataset do not generalize better than unaugmented datasets of similar size and that counterfactual augmentation can hurt performance, yielding models that are less robust to challenge examples. Counterfactual augmentation of natural language understanding data through standard crowdsourcing techniques does not appear to be an effective way of collecting training data and further innovation is required to make this general line of work viable.","url_abs":"https://arxiv.org/abs/2010.04762v1","url_pdf":"https://arxiv.org/pdf/2010.04762v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"counterfactually-augmented-snli-training-data","repo_url":"https://github.com/nyu-mll/CNLI-generalization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2010.04762","atlas_url":"https://app.syntology.ai/?focus=2010.04762","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.04762"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nyu-mll/CNLI-generalization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":15},"by_repo_kind":{"official":{"samples":16,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"61ffc5549a75fe3c","entry":"metadata_collate_fn","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/tasks/core.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/tasks/core.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61ffc5549a75fe3c"}},{"code_sha256_prefix":"e445a3b7d6a58154","entry":"batched_index_select","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/ext/allennlp.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/ext/allennlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e445a3b7d6a58154"}},{"code_sha256_prefix":"eaa6b3cef2ce9444","entry":"compare_dataset_with_metadata","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/caching.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/caching.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eaa6b3cef2ce9444"}},{"code_sha256_prefix":"0f5c37217af75b26","entry":"compare_tensor_tuples","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/caching.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/caching.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f5c37217af75b26"}},{"code_sha256_prefix":"9b5f1ac1298e3440","entry":"complex_backpropagate","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/runner.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b5f1ac1298e3440"}},{"code_sha256_prefix":"2d5bbf46d4d7af68","entry":"convert_to_chunks","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/caching.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/caching.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d5bbf46d4d7af68"}},{"code_sha256_prefix":"507ef2b2ebff6a5c","entry":"flat_collate_fn","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/tasks/core.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/tasks/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"507ef2b2ebff6a5c"}},{"code_sha256_prefix":"cb33d2bf922529ee","entry":"get_label_id_from_data_row","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/tasks/evaluate/core.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/tasks/evaluate/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33d2bf922529ee"}},{"code_sha256_prefix":"42ef1356dacb097d","entry":"get_label_ids","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/tasks/evaluate/core.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/tasks/evaluate/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42ef1356dacb097d"}},{"code_sha256_prefix":"7e8ea392aad26f56","entry":"get_model_and_tokenizer_classes","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/proj/main/export_model.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/proj/main/export_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e8ea392aad26f56"}},{"code_sha256_prefix":"fa0fcf05318dae3d","entry":"init_cuda_from_args","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/initialization.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/initialization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa0fcf05318dae3d"}},{"code_sha256_prefix":"540028e9d806d8a8","entry":"init_seed","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/initialization.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/initialization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"540028e9d806d8a8"}},{"code_sha256_prefix":"16cde5ff8ae0a586","entry":"masked_softmax","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/ext/allennlp.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/ext/allennlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16cde5ff8ae0a586"}},{"code_sha256_prefix":"156059a38968a584","entry":"resolve_is_lower_case","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/model_resolution.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/model_resolution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"156059a38968a584"}},{"code_sha256_prefix":"c7ea7547e7771a4a","entry":"resolve_tokenizer_class","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/shared/model_resolution.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/shared/model_resolution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7ea7547e7771a4a"}},{"code_sha256_prefix":"2f417d486bcad523","entry":"weighted_sum","repo":"nyu-mll/CNLI-generalization","repo_kind":"official","path":"jiant/jiant/ext/allennlp.py","file_url":"https://github.com/nyu-mll/CNLI-generalization/blob/HEAD/jiant/jiant/ext/allennlp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f417d486bcad523"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}