{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/disco-distilling-phrasal-counterfactuals-with","title":"DISCO: Distilling Counterfactuals with Large Language Models","arxiv_id":"2212.10534","date":"2022-12-20","proceeding":null,"authors":["Zeming Chen","Qiyue Gao","Antoine Bosselut","Ashish Sabharwal","Kyle Richardson"],"abstract":"Models trained with counterfactually augmented data learn representations of the causal structure of tasks, enabling robust generalization. However, high-quality counterfactual data is scarce for most tasks and not easily generated at scale. When crowdsourced, such data is typically limited in scale and diversity; when generated using supervised methods, it is computationally expensive to extend to new counterfactual dimensions. In this work, we introduce DISCO (DIStilled COunterfactual Data), a new method for automatically generating high quality counterfactual data at scale. DISCO engineers prompts to generate phrasal perturbations with a large general language model. Then, a task-specific teacher model filters these generations to distill high-quality counterfactual data. While task-agnostic, we apply our pipeline to the task of natural language inference (NLI) and find that on challenging evaluations such as the NLI stress test, comparatively smaller student models trained with DISCO generated counterfactuals are more robust (6% absolute) and generalize better across distributions (2%) compared to models trained without data augmentation. Furthermore, DISCO augmented models are 10% more consistent between counterfactual pairs on three evaluation sets, demonstrating that DISCO augmentation enables models to more reliably learn causal representations. Our repository is available at: https://github.com/eric11eca/disco","url_abs":"https://arxiv.org/abs/2212.10534v3","url_pdf":"https://arxiv.org/pdf/2212.10534v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"disco-distilling-phrasal-counterfactuals-with","repo_url":"https://github.com/eric11eca/disco","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"sentence-classification","task_name":"Sentence Classification"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[{"method_slug":"counterfactuals","method_name":"Counterfactuals"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.10534","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.10534"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eric11eca/disco","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"914b6df319e7ae9e","entry":"get_all","repo":"eric11eca/disco","repo_kind":"official","path":"distiller/db.py","file_url":"https://github.com/eric11eca/disco/blob/HEAD/distiller/db.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"914b6df319e7ae9e"}},{"code_sha256_prefix":"de9b161971667f91","entry":"query","repo":"eric11eca/disco","repo_kind":"official","path":"distiller/db.py","file_url":"https://github.com/eric11eca/disco/blob/HEAD/distiller/db.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de9b161971667f91"}},{"code_sha256_prefix":"56ab26e03a4cb497","entry":"read_json","repo":"eric11eca/disco","repo_kind":"official","path":"distiller/utils.py","file_url":"https://github.com/eric11eca/disco/blob/HEAD/distiller/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56ab26e03a4cb497"}},{"code_sha256_prefix":"19ac4ecaabed84b0","entry":"to_jsonl","repo":"eric11eca/disco","repo_kind":"official","path":"distiller/utils.py","file_url":"https://github.com/eric11eca/disco/blob/HEAD/distiller/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19ac4ecaabed84b0"}},{"code_sha256_prefix":"8cdff0a489b59936","entry":"write_json","repo":"eric11eca/disco","repo_kind":"official","path":"distiller/utils.py","file_url":"https://github.com/eric11eca/disco/blob/HEAD/distiller/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cdff0a489b59936"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}