{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/codes-towards-building-open-source-language","title":"CodeS: Towards Building Open-source Language Models for Text-to-SQL","arxiv_id":"2402.16347","date":"2024-02-26","proceeding":null,"authors":["Haoyang Li","Jing Zhang","Hanbing Liu","Ju Fan","Xiaokang Zhang","Jun Zhu","Renjie Wei","Hongyan Pan","Cuiping Li","Hong Chen"],"abstract":"Language models have shown promising performance on the task of translating natural language questions into SQL queries (Text-to-SQL). However, most of the state-of-the-art (SOTA) approaches rely on powerful yet closed-source large language models (LLMs), such as ChatGPT and GPT-4, which may have the limitations of unclear model architectures, data privacy risks, and expensive inference overheads. To address the limitations, we introduce CodeS, a series of pre-trained language models with parameters ranging from 1B to 15B, specifically designed for the text-to-SQL task. CodeS is a fully open-source language model, which achieves superior accuracy with much smaller parameter sizes. This paper studies the research challenges in building CodeS. To enhance the SQL generation abilities of CodeS, we adopt an incremental pre-training approach using a specifically curated SQL-centric corpus. Based on this, we address the challenges of schema linking and rapid domain adaptation through strategic prompt construction and a bi-directional data augmentation technique. We conduct comprehensive evaluations on multiple datasets, including the widely used Spider benchmark, the newly released BIRD benchmark, robustness-diagnostic benchmarks such as Spider-DK, Spider-Syn, Spider-Realistic, and Dr.Spider, as well as two real-world datasets created for financial and academic applications. The experimental results show that our CodeS achieves new SOTA accuracy and robustness on nearly all challenging text-to-SQL benchmarks.","url_abs":"https://arxiv.org/abs/2402.16347v1","url_pdf":"https://arxiv.org/pdf/2402.16347v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"codes-towards-building-open-source-language","repo_url":"https://github.com/ruckbreasoning/codes","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-to-sql","task_name":"Text to SQL"},{"task_slug":"text-to-sql","task_name":"Text-To-SQL"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16347","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16347"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruckbreasoning/codes","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9c09b83ecd4e3646","entry":"compare_sql","repo":"ruckbreasoning/codes","repo_kind":"official","path":"evaluate_ex.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/evaluate_ex.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c09b83ecd4e3646"}},{"code_sha256_prefix":"fde45273dd27b383","entry":"extract_large_numbers","repo":"ruckbreasoning/codes","repo_kind":"official","path":"prepare_sft_datasets.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/prepare_sft_datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fde45273dd27b383"}},{"code_sha256_prefix":"56e8906d4b412b25","entry":"get_schema","repo":"ruckbreasoning/codes","repo_kind":"official","path":"schema_item_filter.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/schema_item_filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56e8906d4b412b25"}},{"code_sha256_prefix":"8af31230a92da0c1","entry":"get_sequence_length","repo":"ruckbreasoning/codes","repo_kind":"official","path":"schema_item_filter.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/schema_item_filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8af31230a92da0c1"}},{"code_sha256_prefix":"49334501abd461a4","entry":"masked_softmax","repo":"ruckbreasoning/codes","repo_kind":"official","path":"modeling_gpt_bigcode.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/modeling_gpt_bigcode.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49334501abd461a4"}},{"code_sha256_prefix":"93d18eb1da1b928c","entry":"prepare_input_ids_and_attention_mask","repo":"ruckbreasoning/codes","repo_kind":"official","path":"text2sql_few_shot.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/text2sql_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93d18eb1da1b928c"}},{"code_sha256_prefix":"379783e2b5131276","entry":"prepare_inputs_and_labels","repo":"ruckbreasoning/codes","repo_kind":"official","path":"schema_item_filter.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/schema_item_filter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"379783e2b5131276"}},{"code_sha256_prefix":"c1c7de5d97c5b309","entry":"remove_similar_comments","repo":"ruckbreasoning/codes","repo_kind":"official","path":"prepare_sft_datasets.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/prepare_sft_datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1c7de5d97c5b309"}},{"code_sha256_prefix":"4a150066109c1a86","entry":"resume_model_and_optimizer","repo":"ruckbreasoning/codes","repo_kind":"official","path":"train_causal_lm.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/train_causal_lm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a150066109c1a86"}},{"code_sha256_prefix":"c9ee130e5a1966a9","entry":"text2sql_func","repo":"ruckbreasoning/codes","repo_kind":"official","path":"text2sql_zero_shot.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/text2sql_zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9ee130e5a1966a9"}},{"code_sha256_prefix":"38cab2f301010b6d","entry":"upcast_masked_softmax","repo":"ruckbreasoning/codes","repo_kind":"official","path":"modeling_gpt_bigcode.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/modeling_gpt_bigcode.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38cab2f301010b6d"}},{"code_sha256_prefix":"49527b68e554c71a","entry":"upcast_softmax","repo":"ruckbreasoning/codes","repo_kind":"official","path":"modeling_gpt_bigcode.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/modeling_gpt_bigcode.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49527b68e554c71a"}},{"code_sha256_prefix":"e4ab0d16733f055d","entry":"execute_sql","repo":"ruckbreasoning/codes","repo_kind":"official","path":"evaluate_ex.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/evaluate_ex.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4ab0d16733f055d"}},{"code_sha256_prefix":"5d9edf7a65f4c0f4","entry":"extract_skeleton","repo":"ruckbreasoning/codes","repo_kind":"official","path":"text2sql_few_shot.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/text2sql_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d9edf7a65f4c0f4"}},{"code_sha256_prefix":"e459a1d6427bf16a","entry":"prepare","repo":"ruckbreasoning/codes","repo_kind":"official","path":"tokenize_pt_corpus.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/tokenize_pt_corpus.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e459a1d6427bf16a"}},{"code_sha256_prefix":"0f37491b931ec512","entry":"prepare_batch_inputs_and_labels","repo":"ruckbreasoning/codes","repo_kind":"official","path":"train_schema_item_filter.py","file_url":"https://github.com/ruckbreasoning/codes/blob/HEAD/train_schema_item_filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f37491b931ec512"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}