{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spider-a-large-scale-human-labeled-dataset","title":"Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task","arxiv_id":"1809.08887","date":"2018-09-24","proceeding":"EMNLP 2018 10","authors":["Tao Yu","Rui Zhang","Kai Yang","Michihiro Yasunaga","Dongxu Wang","Zifan Li","James Ma","Irene Li","Qingning Yao","Shanelle Roman","Zilin Zhang","Dragomir Radev"],"abstract":"We present Spider, a large-scale, complex and cross-domain semantic parsing\nand text-to-SQL dataset annotated by 11 college students. It consists of 10,181\nquestions and 5,693 unique complex SQL queries on 200 databases with multiple\ntables, covering 138 different domains. We define a new complex and\ncross-domain semantic parsing and text-to-SQL task where different complex SQL\nqueries and databases appear in train and test sets. In this way, the task\nrequires the model to generalize well to both new SQL queries and new database\nschemas. Spider is distinct from most of the previous semantic parsing tasks\nbecause they all use a single database and the exact same programs in the train\nset and the test set. We experiment with various state-of-the-art models and\nthe best model achieves only 12.4% exact matching accuracy on a database split\nsetting. This shows that Spider presents a strong challenge for future\nresearch. Our dataset and task are publicly available at\nhttps://yale-lily.github.io/spider","url_abs":"http://arxiv.org/abs/1809.08887v5","url_pdf":"http://arxiv.org/pdf/1809.08887v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spider-a-large-scale-human-labeled-dataset","repo_url":"https://github.com/taoyds/spider","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"spider-a-large-scale-human-labeled-dataset","repo_url":"https://github.com/ML-KULeuven/deepstochlog-lm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"spider-a-large-scale-human-labeled-dataset","repo_url":"https://github.com/openbotai/nl2sql","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"paper_slug":"spider-a-large-scale-human-labeled-dataset","repo_url":"https://github.com/saparina/text2sql-nlvariation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"spider-a-large-scale-human-labeled-dataset","repo_url":"https://github.com/yandex-research/sparqling-queries","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"spider-a-large-scale-human-labeled-dataset","repo_url":"https://github.com/ygan/spider","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"semantic-parsing","task_name":"Semantic Parsing"},{"task_slug":"text-to-sql","task_name":"Text to SQL"},{"task_slug":"text-to-sql","task_name":"Text-To-SQL"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-parsing-on-spider","task":"Semantic Parsing","dataset":"spider","model":"Exact Set Matching","rank_in_archive_order":10,"of":10,"metrics":{"Accuracy":"19.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1809.08887","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.08887"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/saparina/text2sql-nlvariation","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openbotai/nl2sql","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ML-KULeuven/deepstochlog-lm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ygan/spider","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yandex-research/sparqling-queries","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/taoyds/spider","reach":null}],"summary":{"ran_draft_wrong":2,"ran_violates":3,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":5,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"abcf216a0f054de7","entry":"build_prompt","repo":"ML-KULeuven/deepstochlog-lm","repo_kind":"listed","path":"src/task2/task2.py","file_url":"https://github.com/ML-KULeuven/deepstochlog-lm/blob/HEAD/src/task2/task2.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"abcf216a0f054de7"}},{"code_sha256_prefix":"8231e3fdffa9c8ee","entry":"condition_has_like","repo":"taoyds/spider","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/taoyds/spider/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8231e3fdffa9c8ee"}},{"code_sha256_prefix":"f2d76234528d3b57","entry":"condition_has_or","repo":"taoyds/spider","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/taoyds/spider/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2d76234528d3b57"}},{"code_sha256_prefix":"954c76e256cc7220","entry":"condition_has_sql","repo":"taoyds/spider","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/taoyds/spider/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"954c76e256cc7220"}},{"code_sha256_prefix":"487252da3c521e4a","entry":"get_domain","repo":"ML-KULeuven/deepstochlog-lm","repo_kind":"listed","path":"src/task2/task2.py","file_url":"https://github.com/ML-KULeuven/deepstochlog-lm/blob/HEAD/src/task2/task2.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"487252da3c521e4a"}},{"code_sha256_prefix":"b0825d653a089615","entry":"natsql_idx_to_table2_idx","repo":"saparina/text2sql-nlvariation","repo_kind":"listed","path":"RESDSQL/NatSQL/natsql2sql/natsql2sql.py","file_url":"https://github.com/saparina/text2sql-nlvariation/blob/HEAD/RESDSQL/NatSQL/natsql2sql/natsql2sql.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b0825d653a089615"}},{"code_sha256_prefix":"48b57e02e3f2c83d","entry":"reversed_link_back_col","repo":"saparina/text2sql-nlvariation","repo_kind":"listed","path":"RESDSQL/NatSQL/natsql2sql/natsql2sql.py","file_url":"https://github.com/saparina/text2sql-nlvariation/blob/HEAD/RESDSQL/NatSQL/natsql2sql/natsql2sql.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"48b57e02e3f2c83d"}},{"code_sha256_prefix":"14ad4335d565ea1d","entry":"check_relation","repo":"saparina/text2sql-nlvariation","repo_kind":"listed","path":"RESDSQL/NatSQL/natsql2sql/natsql2sql.py","file_url":"https://github.com/saparina/text2sql-nlvariation/blob/HEAD/RESDSQL/NatSQL/natsql2sql/natsql2sql.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-SA-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"14ad4335d565ea1d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}