{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchclamp-a-benchmark-for-evaluating","title":"BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing","arxiv_id":"2206.10668","date":"2022-06-21","proceeding":"NeurIPS 2023 11","authors":["Subhro Roy","Sam Thomson","Tongfei Chen","Richard Shin","Adam Pauls","Jason Eisner","Benjamin Van Durme"],"abstract":"Recent work has shown that generation from a prompted or fine-tuned language model can perform well at semantic parsing when the output is constrained to be a valid semantic representation. We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Model Parsing, that includes context-free grammars for seven semantic parsing datasets and two syntactic parsing datasets with varied output representations, as well as a constrained decoding interface to generate only valid outputs covered by these grammars. We provide low, medium, and high resource splits for each dataset, allowing accurate comparison of various language models under different data regimes. Our benchmark supports evaluation of language models using prompt-based learning as well as fine-tuning. We benchmark eight language models, including two GPT-3 variants available only through an API. Our experiments show that encoder-decoder pretrained language models can achieve similar performance or surpass state-of-the-art methods for syntactic and semantic parsing when the model output is constrained to be valid.","url_abs":"https://arxiv.org/abs/2206.10668v2","url_pdf":"https://arxiv.org/pdf/2206.10668v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchclamp-a-benchmark-for-evaluating","repo_url":"https://github.com/microsoft/semantic_parsing_with_constrained_lm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"semantic-parsing","task_name":"Semantic Parsing"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.10668","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.10668"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/semantic_parsing_with_constrained_lm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4183422535231a5a","entry":"filter_exp_dict","repo":"microsoft/semantic_parsing_with_constrained_lm","repo_kind":"official","path":"src/semantic_parsing_with_constrained_lm/run_exp.py","file_url":"https://github.com/microsoft/semantic_parsing_with_constrained_lm/blob/HEAD/src/semantic_parsing_with_constrained_lm/run_exp.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4183422535231a5a"}},{"code_sha256_prefix":"dbda3623619056cc","entry":"openai_token_to_bytes","repo":"microsoft/semantic_parsing_with_constrained_lm","repo_kind":"official","path":"src/semantic_parsing_with_constrained_lm/lm_openai_gpt3.py","file_url":"https://github.com/microsoft/semantic_parsing_with_constrained_lm/blob/HEAD/src/semantic_parsing_with_constrained_lm/lm_openai_gpt3.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbda3623619056cc"}},{"code_sha256_prefix":"5a2ae711aef94233","entry":"compute_and_print_fit","repo":"microsoft/semantic_parsing_with_constrained_lm","repo_kind":"official","path":"src/semantic_parsing_with_constrained_lm/fit_max_steps.py","file_url":"https://github.com/microsoft/semantic_parsing_with_constrained_lm/blob/HEAD/src/semantic_parsing_with_constrained_lm/fit_max_steps.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a2ae711aef94233"}},{"code_sha256_prefix":"8a3ce80af5668522","entry":"cross_validation_fit","repo":"microsoft/semantic_parsing_with_constrained_lm","repo_kind":"official","path":"src/semantic_parsing_with_constrained_lm/fit_max_steps.py","file_url":"https://github.com/microsoft/semantic_parsing_with_constrained_lm/blob/HEAD/src/semantic_parsing_with_constrained_lm/fit_max_steps.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a3ce80af5668522"}},{"code_sha256_prefix":"b1defe1f92175d5e","entry":"filter_fit","repo":"microsoft/semantic_parsing_with_constrained_lm","repo_kind":"official","path":"src/semantic_parsing_with_constrained_lm/fit_max_steps.py","file_url":"https://github.com/microsoft/semantic_parsing_with_constrained_lm/blob/HEAD/src/semantic_parsing_with_constrained_lm/fit_max_steps.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1defe1f92175d5e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}