{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coco-controllable-counterfactuals-for-1","title":"CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers","arxiv_id":"2010.12850","date":"2020-10-24","proceeding":"ICLR 2021 1","authors":["Shiyang Li","Semih Yavuz","Kazuma Hashimoto","Jia Li","Tong Niu","Nazneen Rajani","Xifeng Yan","Yingbo Zhou","Caiming Xiong"],"abstract":"Dialogue state trackers have made significant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the held-out conversations is less understood. We propose controllable counterfactuals (CoCo) to bridge this gap and evaluate dialogue state tracking (DST) models on novel scenarios, i.e., would the system successfully tackle the request if the user responded differently but still consistently with the dialogue flow? CoCo leverages turn-level belief states as counterfactual conditionals to produce novel conversation scenarios in two steps: (i) counterfactual goal generation at turn-level by dropping and adding slots followed by replacing slot values, (ii) counterfactual conversation generation that is conditioned on (i) and consistent with the dialogue flow. Evaluating state-of-the-art DST models on MultiWOZ dataset with CoCo-generated counterfactuals results in a significant performance drop of up to 30.8% (from 49.4% to 18.6%) in absolute joint goal accuracy. In comparison, widely used techniques like paraphrasing only affect the accuracy by at most 2%. Human evaluations show that COCO-generated conversations perfectly reflect the underlying user goal with more than 95% accuracy and are as human-like as the original conversations, further strengthening its reliability and promise to be adopted as part of the robustness evaluation of DST models.","url_abs":"https://arxiv.org/abs/2010.12850v3","url_pdf":"https://arxiv.org/pdf/2010.12850v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coco-controllable-counterfactuals-for-1","repo_url":"https://github.com/salesforce/coco-dst","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"coco-controllable-counterfactuals-for-1","repo_url":"https://github.com/wise-east/checkdst","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"dialogue-state-tracking","task_name":"Dialogue State Tracking"},{"task_slug":"multi-domain-dialogue-state-tracking","task_name":"Multi-domain Dialogue State Tracking"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[{"method_slug":"counterfactuals","method_name":"Counterfactuals"},{"method_slug":"dst","method_name":"DST"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2010.12850","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.12850"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salesforce/coco-dst","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wise-east/checkdst","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":5,"ran":1,"ran_violates":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1},"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"440751480098ee7d","entry":"adjust_length_to_model","repo":"salesforce/coco-dst","repo_kind":"official","path":"coco-dst/run_demo.py","file_url":"https://github.com/salesforce/coco-dst/blob/HEAD/coco-dst/run_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"440751480098ee7d"}},{"code_sha256_prefix":"ac5eceaffcbcc5af","entry":"counterfactual_goal_generator","repo":"salesforce/coco-dst","repo_kind":"official","path":"coco-dst/utility/slot_value_ctrl.py","file_url":"https://github.com/salesforce/coco-dst/blob/HEAD/coco-dst/utility/slot_value_ctrl.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ac5eceaffcbcc5af"}},{"code_sha256_prefix":"20284934c6549d67","entry":"extract_slot_from_string","repo":"wise-east/checkdst","repo_kind":"listed","path":"src/checkdst/checkdst_class.py","file_url":"https://github.com/wise-east/checkdst/blob/HEAD/src/checkdst/checkdst_class.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20284934c6549d67"}},{"code_sha256_prefix":"0d250791b826533a","entry":"fix_commonsense","repo":"salesforce/coco-dst","repo_kind":"official","path":"coco-dst/utility/slot_value_ctrl.py","file_url":"https://github.com/salesforce/coco-dst/blob/HEAD/coco-dst/utility/slot_value_ctrl.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0d250791b826533a"}},{"code_sha256_prefix":"e53db5f608c2b832","entry":"gen_time_pair","repo":"salesforce/coco-dst","repo_kind":"official","path":"coco-dst/utility/slot_value_ctrl.py","file_url":"https://github.com/salesforce/coco-dst/blob/HEAD/coco-dst/utility/slot_value_ctrl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e53db5f608c2b832"}},{"code_sha256_prefix":"893a9888cdee1f62","entry":"is_proper_slot_format","repo":"wise-east/checkdst","repo_kind":"listed","path":"src/checkdst/checkdst_class.py","file_url":"https://github.com/wise-east/checkdst/blob/HEAD/src/checkdst/checkdst_class.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"893a9888cdee1f62"}},{"code_sha256_prefix":"7694471c3bd27d3f","entry":"normalize_dial_ids","repo":"wise-east/checkdst","repo_kind":"listed","path":"src/checkdst/checkdst_class.py","file_url":"https://github.com/wise-east/checkdst/blob/HEAD/src/checkdst/checkdst_class.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7694471c3bd27d3f"}},{"code_sha256_prefix":"f23b946820bfadae","entry":"re_match","repo":"salesforce/coco-dst","repo_kind":"official","path":"coco-dst/utility/slot_value_ctrl.py","file_url":"https://github.com/salesforce/coco-dst/blob/HEAD/coco-dst/utility/slot_value_ctrl.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f23b946820bfadae"}},{"code_sha256_prefix":"f769384231a88de1","entry":"update_turn","repo":"salesforce/coco-dst","repo_kind":"official","path":"coco-dst/utility/slot_value_ctrl.py","file_url":"https://github.com/salesforce/coco-dst/blob/HEAD/coco-dst/utility/slot_value_ctrl.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f769384231a88de1"}},{"code_sha256_prefix":"1b22a71e80815e81","entry":"CheckDST","repo":"wise-east/checkdst","repo_kind":"listed","path":"src/checkdst/checkdst_class.py","file_url":"https://github.com/wise-east/checkdst/blob/HEAD/src/checkdst/checkdst_class.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b22a71e80815e81"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}