{"url":"/task/program-synthesis","name":"Program Synthesis","slug":"program-synthesis","description_markdown":"Program synthesis is the process of automatically generating a program or code snippet that satisfies a given specification or set of requirements. This can include generating code from a formal specification, a natural language description, or example inputs and outputs. The primary goal of program synthesis is to minimize human intervention in the coding process, reduce errors, and improve productivity.\r\n\r\nProgram synthesis often involves the use of advanced algorithms, artificial intelligence, and machine learning techniques to search the space of possible programs that meet the given constraints. This process can be guided by a variety of techniques, such as constraint solving, symbolic execution, and genetic algorithms.","categories":[{"name":"Computer Code","url":"/area/computer-code"},{"name":"Reasoning","url":"/area/reasoning"}],"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","slug_source":"archive_url"},"counts":{"papers_tagged":423,"papers_with_code":179,"benchmarks":3,"benchmark_tables_in_archive":3,"benchmark_tables_shown":3,"benchmark_tables_withheld_as_spam":0,"benchmark_definition":"a leaderboard table with at least one row; benchmark_tables_shown also counts the zero-row tables; benchmark_tables_in_archive adds the tables withheld as spam","datasets":5,"subtasks":5,"parent_tasks":0},"benchmarks":[{"leaderboard":"/sota/program-synthesis-on-spoc-testp","slug":"program-synthesis-on-spoc-testp","dataset":"SPoC TestP","dataset_url":"/dataset/spoc","rows_in_archive":2,"metrics":["Success rate @budget 100"],"first_row_in_archive_order":{"model":"DrRepair","paper_title":"Graph-based, Self-Supervised Program Repair from Diagnostic Feedback","paper_url":"/paper/graph-based-self-supervised-program-repair","paper_date":"2020-05-20","arxiv_id":"2005.10636","code_links":[{"title":"michiyasunaga/DrRepair","url":"https://github.com/michiyasunaga/DrRepair"},{"title":"worksheets/0x01838644","url":"https://worksheets.codalab.org/worksheets/0x01838644724a433c932bef4cb5c42fbd"}],"syntology":{"n":6,"n_ran":4,"n_unverified":2,"n_pointer_only":0}}},{"leaderboard":"/sota/program-synthesis-on-spoc-testw","slug":"program-synthesis-on-spoc-testw","dataset":"SPoC TestW","dataset_url":"/dataset/spoc","rows_in_archive":2,"metrics":["Success rate @budget 100"],"first_row_in_archive_order":{"model":"DrRepair","paper_title":"Graph-based, Self-Supervised Program Repair from Diagnostic Feedback","paper_url":"/paper/graph-based-self-supervised-program-repair","paper_date":"2020-05-20","arxiv_id":"2005.10636","code_links":[{"title":"michiyasunaga/DrRepair","url":"https://github.com/michiyasunaga/DrRepair"},{"title":"worksheets/0x01838644","url":"https://worksheets.codalab.org/worksheets/0x01838644724a433c932bef4cb5c42fbd"}],"syntology":{"n":6,"n_ran":4,"n_unverified":2,"n_pointer_only":0}}},{"leaderboard":"/sota/program-synthesis-on-algolisp","slug":"program-synthesis-on-algolisp","dataset":"AlgoLisp","dataset_url":null,"rows_in_archive":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"CodeTrans-MT-TF-Small","paper_title":"CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computing","paper_url":"/paper/codetrans-towards-cracking-the-language-of","paper_date":"2021-04-06","arxiv_id":"2104.02443","code_links":[{"title":"agemagician/CodeTrans","url":"https://github.com/agemagician/CodeTrans"}],"syntology":null}}],"datasets":[{"url":"/dataset/concode","name":"CONCODE","full_name":"","num_papers_in_archive":46},{"url":"/dataset/sketchgraphs","name":"SketchGraphs","full_name":null,"num_papers_in_archive":16},{"url":"/dataset/spoc","name":"SPoC","full_name":"Pseudocode-to-Code","num_papers_in_archive":16},{"url":"/dataset/xcodeeval","name":"xCodeEval","full_name":"xCodeEval","num_papers_in_archive":15},{"url":"/dataset/psb2","name":"PSB2","full_name":"The Second Program Synthesis Benchmark Suite","num_papers_in_archive":1}],"subtasks":[{"url":"/task/enumerative-search","name":"Enumerative Search"},{"url":"/task/program-repair","name":"Program Repair"},{"url":"/task/sql-synthesis","name":"SQL Synthesis"},{"url":"/task/type-prediction","name":"Type prediction"},{"url":"/task/value-prediction","name":"Value prediction"}],"parent_tasks":[],"papers":{"order":"repositories listed in the archive (desc), then date (desc); the archive holds no stars","population":"papers tagged with this task that list at least one repository in the archive","shown":30,"of":179,"tagged_in_all":423,"items":[{"url":"/paper/a-conversational-paradigm-for-program","title":"CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis","date":"2022-03-25","arxiv_id":"2203.13474","repositories_listed":8,"syntology":{"n":10,"n_ran":10,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/large-language-models-are-human-level-prompt","title":"Large Language Models Are Human-Level Prompt Engineers","date":"2022-11-03","arxiv_id":"2211.01910","repositories_listed":5,"syntology":{"n":1,"n_ran":1,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/xcodeeval-a-large-scale-multilingual","title":"xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval","date":"2023-03-06","arxiv_id":"2303.03004","repositories_listed":4,"syntology":{"n":6,"n_ran":1,"n_unverified":5,"n_pointer_only":0}},{"url":"/paper/memory-augmented-policy-optimization-for","title":"Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing","date":"2018-07-06","arxiv_id":"1807.02322","repositories_listed":4,"syntology":{"n":15,"n_ran":0,"n_unverified":15,"n_pointer_only":0}},{"url":"/paper/neural-program-synthesis-with-priority-queue","title":"Neural Program Synthesis with Priority Queue Training","date":"2018-01-10","arxiv_id":"1801.03526","repositories_listed":4,"syntology":null},{"url":"/paper/incoder-a-generative-model-for-code-infilling","title":"InCoder: A Generative Model for Code Infilling and Synthesis","date":"2022-04-12","arxiv_id":"2204.05999","repositories_listed":3,"syntology":{"n":10,"n_ran":6,"n_unverified":4,"n_pointer_only":2}},{"url":"/paper/program-synthesis-for-the-oeis","title":"Learning Program Synthesis for Integer Sequences from Scratch","date":"2022-02-24","arxiv_id":"2202.11908","repositories_listed":3,"syntology":null},{"url":"/paper/programming-puzzles","title":"Programming Puzzles","date":"2021-06-10","arxiv_id":"2106.05784","repositories_listed":3,"syntology":null},{"url":"/paper/dreamcoder-growing-generalizable","title":"DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning","date":"2020-06-15","arxiv_id":"2006.08381","repositories_listed":3,"syntology":{"n":11,"n_ran":1,"n_unverified":10,"n_pointer_only":11}},{"url":"/paper/robustfill-neural-program-learning-under","title":"RobustFill: Neural Program Learning under Noisy I/O","date":"2017-03-21","arxiv_id":"1703.07469","repositories_listed":3,"syntology":{"n":8,"n_ran":2,"n_unverified":6,"n_pointer_only":1}},{"url":"/paper/deepcoder-learning-to-write-programs","title":"DeepCoder: Learning to Write Programs","date":"2016-11-07","arxiv_id":"1611.01989","repositories_listed":3,"syntology":{"n":8,"n_ran":3,"n_unverified":5,"n_pointer_only":3}},{"url":"/paper/clever-a-curated-benchmark-for-formally","title":"CLEVER: A Curated Benchmark for Formally Verified Code Generation","date":"2025-05-20","arxiv_id":"2505.13938","repositories_listed":2,"syntology":null},{"url":"/paper/tikzero-zero-shot-text-guided-graphics","title":"TikZero: Zero-Shot Text-Guided Graphics Program Synthesis","date":"2025-03-14","arxiv_id":"2503.11509","repositories_listed":2,"syntology":null},{"url":"/paper/mapcoder-multi-agent-code-generation-for","title":"MapCoder: Multi-Agent Code Generation for Competitive Problem Solving","date":"2024-05-18","arxiv_id":"2405.11403","repositories_listed":2,"syntology":{"n":10,"n_ran":0,"n_unverified":10,"n_pointer_only":0}},{"url":"/paper/bring-your-own-kg-self-supervised-program","title":"Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA","date":"2023-11-14","arxiv_id":"2311.07850","repositories_listed":2,"syntology":null},{"url":"/paper/knowledge-driven-robot-program-synthesis-from","title":"Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations","date":"2023-06-05","arxiv_id":"2306.02739","repositories_listed":2,"syntology":null},{"url":"/paper/codegen2-lessons-for-training-llms-on","title":"CodeGen2: Lessons for Training LLMs on Programming and Natural Languages","date":"2023-05-03","arxiv_id":"2305.02309","repositories_listed":2,"syntology":null},{"url":"/paper/learning-programs-with-magic-values","title":"Learning programs with magic values","date":"2022-08-05","arxiv_id":"2208.03238","repositories_listed":2,"syntology":null},{"url":"/paper/coderl-mastering-code-generation-through","title":"CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning","date":"2022-07-05","arxiv_id":"2207.01780","repositories_listed":2,"syntology":{"n":1,"n_ran":1,"n_unverified":0,"n_pointer_only":0}},{"url":"/paper/communicating-natural-programs-to-humans-and","title":"Communicating Natural Programs to Humans and Machines","date":"2021-06-15","arxiv_id":"2106.07824","repositories_listed":2,"syntology":null},{"url":"/paper/graph-based-self-supervised-program-repair","title":"Graph-based, Self-Supervised Program Repair from Diagnostic Feedback","date":"2020-05-20","arxiv_id":"2005.10636","repositories_listed":2,"syntology":{"n":6,"n_ran":4,"n_unverified":2,"n_pointer_only":0}},{"url":"/paper/tf-coder-program-synthesis-for-tensor","title":"TF-Coder: Program Synthesis for Tensor Manipulations","date":"2020-03-19","arxiv_id":"2003.09040","repositories_listed":2,"syntology":{"n":14,"n_ran":0,"n_unverified":14,"n_pointer_only":0}},{"url":"/paper/improving-molecular-design-by-stochastic","title":"Improving Molecular Design by Stochastic Iterative Target Augmentation","date":"2020-02-11","arxiv_id":"2002.04720","repositories_listed":2,"syntology":{"n":8,"n_ran":0,"n_unverified":8,"n_pointer_only":0}},{"url":"/paper/natural-to-formal-language-generation-using-1","title":"Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations","date":"2019-10-05","arxiv_id":"1910.02339","repositories_listed":2,"syntology":{"n":3,"n_ran":3,"n_unverified":0,"n_pointer_only":3}},{"url":"/paper/houdini-lifelong-learning-as-program","title":"HOUDINI: Lifelong Learning as Program Synthesis","date":"2018-03-31","arxiv_id":"1804.00218","repositories_listed":2,"syntology":{"n":3,"n_ran":2,"n_unverified":1,"n_pointer_only":3}},{"url":"/paper/transductively-informed-inductive-program","title":"Transductively Informed Inductive Program Synthesis","date":"2025-05-20","arxiv_id":"2505.14744","repositories_listed":1,"syntology":null},{"url":"/paper/2505-10819","title":"PoE-World: Compositional World Modeling with Products of Programmatic Experts","date":"2025-05-16","arxiv_id":"2505.10819","repositories_listed":1,"syntology":null},{"url":"/paper/enhancing-code-generation-via-bidirectional","title":"Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding","date":"2025-05-12","arxiv_id":"2505.07768","repositories_listed":1,"syntology":null},{"url":"/paper/rewriting-pre-training-data-boosts-llm","title":"Rewriting Pre-Training Data Boosts LLM Performance in Math and Code","date":"2025-05-05","arxiv_id":"2505.02881","repositories_listed":1,"syntology":{"n":2,"n_ran":2,"n_unverified":0,"n_pointer_only":2}},{"url":"/paper/osvbench-benchmarking-llms-on-specification","title":"OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification","date":"2025-04-29","arxiv_id":"2504.20964","repositories_listed":1,"syntology":{"n":1,"n_ran":1,"n_unverified":0,"n_pointer_only":1}}],"syntology_records":17,"syntology_note":"a paper without a record is not a recorded non-run: it may lack an arXiv id or simply be absent from the graph layer"},"description_links":{"kept":0,"unwrapped_to_text":0,"bare_urls_linked":0,"relative_images_dropped":0,"rule":"internal links are kept only when the target slug exists in the catalog"},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per-sample execution status on synthesized fixtures ('ran N of M samples'); not a correctness claim and not a ranking signal.","status_vocabulary":{"ran_honours":"ran, honoured the contract we drafted","ran_violates":"ran, violated the contract we drafted","ran_draft_wrong":"ran; our contract draft was wrong, not the code","ran_fixture":"ran; our fixture could not drive it","ran":"ran on a synthesized input","unverified":"unverified (harvested, no recorded run)"}},"not_shown":{"libraries":"the archive has no per-task library table","trend_sparklines":"the Trend column of the benchmarks table was a rendered image; it is not in the archive","social_and_latest_sorts":"stars and social signals are not in the archive"}}