{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/binding-language-models-in-symbolic-languages","title":"Binding Language Models in Symbolic Languages","arxiv_id":"2210.02875","date":"2022-10-06","proceeding":null,"authors":["Zhoujun Cheng","Tianbao Xie","Peng Shi","Chengzu Li","Rahul Nadkarni","Yushi Hu","Caiming Xiong","Dragomir Radev","Mari Ostendorf","Luke Zettlemoyer","Noah A. Smith","Tao Yu"],"abstract":"Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations. Specifically, we employ GPT-3 Codex as the LM. In the parsing stage, with only a few in-context exemplars, Codex is able to identify the part of the task input that cannot be answerable by the original programming language, correctly generate API calls to prompt Codex to solve the unanswerable part, and identify where to place the API calls while being compatible with the original grammar. In the execution stage, Codex can perform versatile functionalities (e.g., commonsense QA, information extraction) given proper prompts in the API calls. Binder achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, with explicit output programs that benefit human debugging. Note that previous best systems are all finetuned on tens of thousands of task-specific samples, while Binder only uses dozens of annotations as in-context exemplars without any training. Our code is available at https://github.com/HKUNLP/Binder .","url_abs":"https://arxiv.org/abs/2210.02875v2","url_pdf":"https://arxiv.org/pdf/2210.02875v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"binding-language-models-in-symbolic-languages","repo_url":"https://github.com/hkunlp/binder","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"binding-language-models-in-symbolic-languages","repo_url":"https://github.com/nikhilsab/h-star","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"binding-language-models-in-symbolic-languages","repo_url":"https://github.com/parkervg/blendsql","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"binding-language-models-in-symbolic-languages","repo_url":"https://github.com/xlang-ai/binder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"semantic-parsing","task_name":"Semantic Parsing"},{"task_slug":"table-based-fact-verification","task_name":"Table-based Fact Verification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-parsing-on-wikitablequestions","task":"Semantic Parsing","dataset":"WikiTableQuestions","model":"Binder","rank_in_archive_order":13,"of":22,"metrics":{"Accuracy (Dev)":"65.0","Accuracy (Test)":"64.6"},"uses_additional_data":false},{"leaderboard":"/sota/table-based-fact-verification-on-tabfact","task":"Table-based Fact Verification","dataset":"TabFact","model":"Binder","rank_in_archive_order":5,"of":15,"metrics":{"Test":"86.0","Val":"-"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.02875","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.02875"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hkunlp/binder","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/parkervg/blendsql","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xlang-ai/binder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nikhilsab/h-star","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"listed":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"68dd94239e84d0ba","entry":"convert_type","repo":"xlang-ai/binder","repo_kind":"listed","path":"nsql/parser.py","file_url":"https://github.com/xlang-ai/binder/blob/HEAD/nsql/parser.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68dd94239e84d0ba"}},{"code_sha256_prefix":"59b8d272e4121f4a","entry":"get_cfg_tree","repo":"xlang-ai/binder","repo_kind":"listed","path":"nsql/parser.py","file_url":"https://github.com/xlang-ai/binder/blob/HEAD/nsql/parser.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59b8d272e4121f4a"}},{"code_sha256_prefix":"a2bdae85fa671acc","entry":"TreeNode","repo":"xlang-ai/binder","repo_kind":"listed","path":"nsql/parser.py","file_url":"https://github.com/xlang-ai/binder/blob/HEAD/nsql/parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2bdae85fa671acc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}