{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cosql-a-conversational-text-to-sql-challenge","title":"CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases","arxiv_id":"1909.05378","date":"2019-09-11","proceeding":"IJCNLP 2019 11","authors":["Tao Yu","Rui Zhang","He Yang Er","Suyi Li","Eric Xue","Bo Pang","Xi Victoria Lin","Yi Chern Tan","Tianze Shi","Zihan Li","Youxuan Jiang","Michihiro Yasunaga","Sungrok Shim","Tao Chen","Alexander Fabbri","Zifan Li","Luyao Chen","Yuwen Zhang","Shreya Dixit","Vincent Zhang","Caiming Xiong","Richard Socher","Walter S. Lasecki","Dragomir Radev"],"abstract":"We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.","url_abs":"https://arxiv.org/abs/1909.05378v1","url_pdf":"https://arxiv.org/pdf/1909.05378v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cosql-a-conversational-text-to-sql-challenge","repo_url":"https://github.com/ryanzhumich/editsql","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"cosql-a-conversational-text-to-sql-challenge","repo_url":"https://github.com/amolk/editsql","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"cosql-a-conversational-text-to-sql-challenge","repo_url":"https://github.com/ryanzhumich/sparc_atis_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dialogue-state-tracking","task_name":"Dialogue State Tracking"},{"task_slug":"response-generation","task_name":"Response Generation"},{"task_slug":"text-to-sql","task_name":"Text to SQL"},{"task_slug":"text-to-sql","task_name":"Text-To-SQL"}],"methods":[],"datasets_introduced":[{"slug":"cosql","name":"CoSQL","full_name":"Conversational Text-to-SQL Challenge"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/dialogue-state-tracking-on-cosql","task":"Dialogue State Tracking","dataset":"CoSQL","model":"SyntaxSQL-con","rank_in_archive_order":8,"of":9,"metrics":{"interaction match accuracy":"2.2","question match accuracy":"14.1"},"uses_additional_data":false},{"leaderboard":"/sota/dialogue-state-tracking-on-cosql","task":"Dialogue State Tracking","dataset":"CoSQL","model":"CD-Seq2seq","rank_in_archive_order":9,"of":9,"metrics":{"interaction match accuracy":"2.6","question match accuracy":"13.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1909.05378","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.05378"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ryanzhumich/sparc_atis_pytorch","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amolk/editsql","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ryanzhumich/editsql","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"listed":{"samples":11,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0bff7fccef337b38","entry":"encode_snippets_with_states","repo":"amolk/editsql","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bff7fccef337b38"}},{"code_sha256_prefix":"7e2d038f758acc4a","entry":"compute_loss","repo":"amolk/editsql","repo_kind":"listed","path":"model/torch_utils.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/torch_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e2d038f758acc4a"}},{"code_sha256_prefix":"c59f3f02b15d3770","entry":"flatten_distribution","repo":"amolk/editsql","repo_kind":"listed","path":"model/decoder.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/decoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c59f3f02b15d3770"}},{"code_sha256_prefix":"8a55b7ef1c467cea","entry":"gen_l_hpu","repo":"amolk/editsql","repo_kind":"listed","path":"model/utils_bert.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/utils_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a55b7ef1c467cea"}},{"code_sha256_prefix":"0ef533abd86a2b26","entry":"generate_inputs","repo":"amolk/editsql","repo_kind":"listed","path":"model/utils_bert.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/utils_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ef533abd86a2b26"}},{"code_sha256_prefix":"9ac04740fba8c9cf","entry":"get_bert","repo":"amolk/editsql","repo_kind":"listed","path":"model/utils_bert.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/utils_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ac04740fba8c9cf"}},{"code_sha256_prefix":"03149eb9ca360505","entry":"get_seq_from_scores","repo":"amolk/editsql","repo_kind":"listed","path":"model/torch_utils.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/torch_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03149eb9ca360505"}},{"code_sha256_prefix":"4f3f1f5678f97059","entry":"linear_layer","repo":"amolk/editsql","repo_kind":"listed","path":"model/torch_utils.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/torch_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f3f1f5678f97059"}},{"code_sha256_prefix":"004b8e1db7db5c18","entry":"score_query_tokens","repo":"amolk/editsql","repo_kind":"listed","path":"model/token_predictor.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/token_predictor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"004b8e1db7db5c18"}},{"code_sha256_prefix":"c3efa7d8a524d46d","entry":"score_schema_tokens","repo":"amolk/editsql","repo_kind":"listed","path":"model/token_predictor.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/token_predictor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3efa7d8a524d46d"}},{"code_sha256_prefix":"26a5daacb4d2f144","entry":"score_snippets","repo":"amolk/editsql","repo_kind":"listed","path":"model/token_predictor.py","file_url":"https://github.com/amolk/editsql/blob/HEAD/model/token_predictor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26a5daacb4d2f144"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}