{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chess-contextual-harnessing-for-efficient-sql","title":"CHESS: Contextual Harnessing for Efficient SQL Synthesis","arxiv_id":"2405.16755","date":"2024-05-27","proceeding":null,"authors":["Shayan Talaei","Mohammadreza Pourreza","Yu-Chen Chang","Azalia Mirhoseini","Amin Saberi"],"abstract":"Translating natural language questions into SQL queries, known as text-to-SQL, is a long-standing research problem. Effective text-to-SQL synthesis can become very challenging due to (i) the extensive size of database catalogs (descriptions of tables and their columns) and database values, (ii) reasoning over large database schemas, (iii) ensuring the functional validity of the generated queries, and (iv) navigating the ambiguities of natural language questions. We introduce CHESS, a Large Language Model (LLM) based multi-agent framework for efficient and scalable SQL synthesis, comprising four specialized agents, each targeting one of the aforementioned challenges: the Information Retriever (IR) extracts relevant data, the Schema Selector (SS) prunes large schemas, the Candidate Generator (CG) generates high-quality candidates and refines queries iteratively, and the Unit Tester (UT) validates queries through LLM-based natural language unit tests. Our framework offers configurable features that adapt to various deployment constraints, including 1) Supporting industrial-scale databases: leveraging the Schema Selector agent, CHESS efficiently narrows down very large database schemas into manageable sub-schemas, boosting system accuracy by approximately $2\\%$ and reducing the number of LLM tokens by $\\times 5$. 2) State-of-the-Art privacy-preserving performance: Among the methods using open-source models, CHESS achieves state-of-the-art performance, resulting in a high-performing, privacy-preserving system suitable for industrial deployment. 3) Scalablity with additional compute budget: In settings with high computational budgets, CHESS achieves $71.10\\%$ accuracy on the BIRD test set, within $2\\%$ of the leading proprietary method, while requiring approximately $83\\%$ fewer LLM calls.","url_abs":"https://arxiv.org/abs/2405.16755v3","url_pdf":"https://arxiv.org/pdf/2405.16755v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chess-contextual-harnessing-for-efficient-sql","repo_url":"https://github.com/shayantalaei/chess","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"chess-contextual-harnessing-for-efficient-sql","repo_url":"https://github.com/yeounoh/lc_nl2sql","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"tasks":[{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"privacy-preserving","task_name":"Privacy Preserving"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"sql-synthesis","task_name":"SQL Synthesis"},{"task_slug":"text-to-sql","task_name":"Text to SQL"},{"task_slug":"text-to-sql","task_name":"Text-To-SQL"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-to-sql-on-bird-big-bench-for-large-scale","task":"Text-To-SQL","dataset":"BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation)","model":"CHESS","rank_in_archive_order":13,"of":41,"metrics":{"Execution Accuracy % (Dev)":"65","Execution Accuracy % (Test)":"66.69"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.16755","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16755"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shayantalaei/chess","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yeounoh/lc_nl2sql","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ad3387002d9f4290","entry":"create_smaller_db","repo":"shayantalaei/chess","repo_kind":"official","path":"src/database_utils/execution.py","file_url":"https://github.com/shayantalaei/chess/blob/HEAD/src/database_utils/execution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad3387002d9f4290"}},{"code_sha256_prefix":"23a00a780614c4eb","entry":"execute_sql","repo":"shayantalaei/chess","repo_kind":"official","path":"src/database_utils/execution.py","file_url":"https://github.com/shayantalaei/chess/blob/HEAD/src/database_utils/execution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23a00a780614c4eb"}},{"code_sha256_prefix":"fe37954301139f99","entry":"get_primary_keys","repo":"shayantalaei/chess","repo_kind":"official","path":"src/database_utils/schema.py","file_url":"https://github.com/shayantalaei/chess/blob/HEAD/src/database_utils/schema.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe37954301139f99"}},{"code_sha256_prefix":"11021b013fcea6fd","entry":"ordered_concurrent_function_calls","repo":"shayantalaei/chess","repo_kind":"official","path":"src/threading_utils.py","file_url":"https://github.com/shayantalaei/chess/blob/HEAD/src/threading_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11021b013fcea6fd"}},{"code_sha256_prefix":"deb7dcb22098a379","entry":"receive_data_in_chunks","repo":"shayantalaei/chess","repo_kind":"official","path":"src/runner/database_manager.py","file_url":"https://github.com/shayantalaei/chess/blob/HEAD/src/runner/database_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"deb7dcb22098a379"}},{"code_sha256_prefix":"4f33589835015c45","entry":"subprocess_sql_executor","repo":"shayantalaei/chess","repo_kind":"official","path":"src/database_utils/execution.py","file_url":"https://github.com/shayantalaei/chess/blob/HEAD/src/database_utils/execution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4f33589835015c45"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}