{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lever-learning-to-verify-language-to-code","title":"LEVER: Learning to Verify Language-to-Code Generation with Execution","arxiv_id":"2302.08468","date":"2023-02-16","proceeding":null,"authors":["Ansong Ni","Srini Iyer","Dragomir Radev","Ves Stoyanov","Wen-tau Yih","Sida I. Wang","Xi Victoria Lin"],"abstract":"The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine LLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the LLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.","url_abs":"https://arxiv.org/abs/2302.08468v3","url_pdf":"https://arxiv.org/pdf/2302.08468v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lever-learning-to-verify-language-to-code","repo_url":"https://github.com/niansong1996/lever","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"math","task_name":"Math"},{"task_slug":"reranking","task_name":"Reranking"},{"task_slug":"semantic-parsing","task_name":"Semantic Parsing"},{"task_slug":"text-to-sql","task_name":"Text-To-SQL"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"code-davinci-002 175B (LEVER, 8-shot)","rank_in_archive_order":50,"of":164,"metrics":{"Accuracy":"84.5","Parameters (Billion)":"175"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"code-davinci-002 175B + LEVER","rank_in_archive_order":33,"of":99,"metrics":{"Accuracy":"68.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-parsing-on-wikitablequestions","task":"Semantic Parsing","dataset":"WikiTableQuestions","model":"LEVER","rank_in_archive_order":11,"of":22,"metrics":{"Accuracy (Dev)":"64.6","Accuracy (Test)":"65.8"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-parsing-on-spider","task":"Semantic Parsing","dataset":"spider","model":"code-davinci-002 175B (LEVER)","rank_in_archive_order":2,"of":10,"metrics":{"Accuracy":"81.9"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-sql-on-spider","task":"Text-To-SQL","dataset":"spider","model":"code-davinci-002 175B (LEVER)","rank_in_archive_order":17,"of":20,"metrics":{"Execution Accuracy (Dev)":"81.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.08468","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.08468"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/niansong1996/lever","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":5,"unverified":16},"by_repo_kind":{"official":{"samples":22,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e18607dae38b30e2","entry":"is_number","repo":"niansong1996/lever","repo_kind":"official","path":"execution/wtq_eval.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/wtq_eval.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e18607dae38b30e2"}},{"code_sha256_prefix":"2178b95c919ae420","entry":"join_tokens","repo":"niansong1996/lever","repo_kind":"official","path":"execution/parse.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/parse.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2178b95c919ae420"}},{"code_sha256_prefix":"f23d6ff3772963bc","entry":"permute_tuple","repo":"niansong1996/lever","repo_kind":"official","path":"execution/spider_official_exec_match.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/spider_official_exec_match.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f23d6ff3772963bc"}},{"code_sha256_prefix":"58e472cff9780d63","entry":"postprocess","repo":"niansong1996/lever","repo_kind":"official","path":"execution/parse.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/parse.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58e472cff9780d63"}},{"code_sha256_prefix":"a40510ba6a37f798","entry":"quick_rej","repo":"niansong1996/lever","repo_kind":"official","path":"execution/spider_official_exec_match.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/spider_official_exec_match.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a40510ba6a37f798"}},{"code_sha256_prefix":"afc8ef89576ef2aa","entry":"unorder_row","repo":"niansong1996/lever","repo_kind":"official","path":"execution/spider_official_exec_match.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/spider_official_exec_match.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afc8ef89576ef2aa"}},{"code_sha256_prefix":"7e87fa354c8a130c","entry":"byte_idx_to_char_idx","repo":"niansong1996/lever","repo_kind":"official","path":"execution/program_tracing.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/program_tracing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e87fa354c8a130c"}},{"code_sha256_prefix":"5601a3dc0fc200e8","entry":"canonicalize_var_dict","repo":"niansong1996/lever","repo_kind":"official","path":"execution/safe_execution_util.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/safe_execution_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5601a3dc0fc200e8"}},{"code_sha256_prefix":"51262b43d9045912","entry":"direct_categorize","repo":"niansong1996/lever","repo_kind":"official","path":"execution/spider_execution.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/spider_execution.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51262b43d9045912"}},{"code_sha256_prefix":"3a39b10c8a429dc8","entry":"estimate_pass_at_k","repo":"niansong1996/lever","repo_kind":"official","path":"few_shot/codex.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/few_shot/codex.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a39b10c8a429dc8"}},{"code_sha256_prefix":"0e6bb17163d0c70b","entry":"execute","repo":"niansong1996/lever","repo_kind":"official","path":"execution/safe_execution_util.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/safe_execution_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e6bb17163d0c70b"}},{"code_sha256_prefix":"5d30efa67a78bf91","entry":"normalize","repo":"niansong1996/lever","repo_kind":"official","path":"execution/wtq_eval.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/wtq_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d30efa67a78bf91"}},{"code_sha256_prefix":"73683dd2b9714a13","entry":"openai_call","repo":"niansong1996/lever","repo_kind":"official","path":"few_shot/codex.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/few_shot/codex.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73683dd2b9714a13"}},{"code_sha256_prefix":"d4c44c1affe7cfe5","entry":"pd_df_from_dict","repo":"niansong1996/lever","repo_kind":"official","path":"execution/spider_execution.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/spider_execution.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4c44c1affe7cfe5"}},{"code_sha256_prefix":"96389cb2ab61121a","entry":"pd_df_to_dict","repo":"niansong1996/lever","repo_kind":"official","path":"execution/spider_execution.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/spider_execution.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96389cb2ab61121a"}},{"code_sha256_prefix":"48ddf72e21e4525b","entry":"python_program_len","repo":"niansong1996/lever","repo_kind":"official","path":"execution/executors.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/executors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48ddf72e21e4525b"}},{"code_sha256_prefix":"acdd63b83770726f","entry":"saved_promptify_mathqa","repo":"niansong1996/lever","repo_kind":"official","path":"finetuning/lightning_modules/datasets/mathqa_reader.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/finetuning/lightning_modules/datasets/mathqa_reader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acdd63b83770726f"}},{"code_sha256_prefix":"f7dbd2bcc2e00ee0","entry":"simple_canonicalize_var_dict","repo":"niansong1996/lever","repo_kind":"official","path":"execution/safe_execution_util.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/safe_execution_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7dbd2bcc2e00ee0"}},{"code_sha256_prefix":"53b15e547590af1b","entry":"sql_program_len","repo":"niansong1996/lever","repo_kind":"official","path":"execution/executors.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/executors.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53b15e547590af1b"}},{"code_sha256_prefix":"4269741e408c7847","entry":"state_answer_only_func","repo":"niansong1996/lever","repo_kind":"official","path":"finetuning/lightning_modules/datasets/mathqa_reader.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/finetuning/lightning_modules/datasets/mathqa_reader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4269741e408c7847"}},{"code_sha256_prefix":"bc02ec961672d267","entry":"state_simple_str_func","repo":"niansong1996/lever","repo_kind":"official","path":"finetuning/lightning_modules/datasets/mathqa_reader.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/finetuning/lightning_modules/datasets/mathqa_reader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc02ec961672d267"}},{"code_sha256_prefix":"efd6f533b8606dca","entry":"to_value","repo":"niansong1996/lever","repo_kind":"official","path":"execution/wtq_eval.py","file_url":"https://github.com/niansong1996/lever/blob/HEAD/execution/wtq_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efd6f533b8606dca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}