Program Synthesis

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

5 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

5 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 179 papers with code (423 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
  25 Mar 2022 8 repositories listed Syntology ran 10 of 10 samples · 0 unverified
  To democratize this, we train and release a family of large language models up to 16.
• Large Language Models Are Human-Level Prompt Engineers
  3 Nov 2022 5 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers.
• xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
  6 Mar 2023 4 repositories listed Syntology ran 1 of 6 samples · 5 unverified
  Recently, pre-trained large language models (LLMs) have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving…
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
  6 Jul 2018 4 repositories listed Syntology ran 0 of 15 samples · 15 unverified
  We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimate.
• Neural Program Synthesis with Priority Queue Training
  10 Jan 2018 4 repositories listed
  Models and examples built with TensorFlow
• InCoder: A Generative Model for Code Infilling and Synthesis
  12 Apr 2022 3 repositories listed Syntology ran 6 of 10 samples · 4 unverified · 2 pointer-only (licence)
  Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming.
• Learning Program Synthesis for Integer Sequences from Scratch
  24 Feb 2022 3 repositories listed
  We present a self-learning approach for synthesizing programs from integer sequences.
• Programming Puzzles
  10 Jun 2021 3 repositories listed
  The dataset is comprehensive in that it spans problems of a range of difficulties and domains, ranging from trivial string manipulation problems, to classic programming puzzles (e.
• DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
  15 Jun 2020 3 repositories listed Syntology ran 1 of 11 samples · 10 unverified · 11 pointer-only (licence)
  It builds expertise by creating programming languages for expressing domain concepts, together with neural networks to guide the search for programs within these languages.
• RobustFill: Neural Program Learning under Noisy I/O
  21 Mar 2017 3 repositories listed Syntology ran 2 of 8 samples · 6 unverified · 1 pointer-only (licence)
  Recently, two competing approaches for automatic program learning have received significant attention: (1) neural program synthesis, where a neural network is conditioned on input/output (I/O) examples and learns to…
• DeepCoder: Learning to Write Programs
  7 Nov 2016 3 repositories listed Syntology ran 3 of 8 samples · 5 unverified · 3 pointer-only (licence)
  We develop a first line of attack for solving programming competition-style problems from input-output examples using deep learning.
• CLEVER: A Curated Benchmark for Formally Verified Code Generation
  20 May 2025 2 repositories listed
  We introduce CLEVER, a high-quality, curated benchmark of 161 problems for end-to-end verified code generation in Lean.
• TikZero: Zero-Shot Text-Guided Graphics Program Synthesis
  14 Mar 2025 2 repositories listed
  Meanwhile, large amounts of unaligned graphics programs and captioned raster images are more readily available.
• MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
  18 May 2024 2 repositories listed Syntology ran 0 of 10 samples · 10 unverified
  In this paper, we introduce a new approach to code generation tasks leveraging multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers.
• Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA
  14 Nov 2023 2 repositories listed
  We present BYOKG, a universal question-answering (QA) system that can operate on any knowledge graph (KG), requires no human-annotated training data, and can be ready to use within a day -- attributes that are…
• Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations
  5 Jun 2023 2 repositories listed
  Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks.
• CodeGen2: Lessons for Training LLMs on Programming and Natural Languages
  3 May 2023 2 repositories listed
  In this study, we attempt to render the training of LLMs for program synthesis more efficient by unifying four key components: (1) model architectures, (2) learning methods, (3) infill sampling, and, (4) data…
• Learning programs with magic values
  5 Aug 2022 2 repositories listed
  A magic value in a program is a constant symbol that is essential for the execution of the program but has no clear explanation for its choice.
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
  5 Jul 2022 2 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  To address the limitations, we propose "CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL).
• Communicating Natural Programs to Humans and Machines
  15 Jun 2021 2 repositories listed
  We present LARC, the \textit{Language-complete ARC}: a collection of natural language descriptions by a group of human participants who instruct each other on how to solve ARC tasks using language alone, which contains…
• Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
  20 May 2020 2 repositories listed Syntology ran 4 of 6 samples · 2 unverified
  Second, we present a self-supervised learning paradigm for program repair that leverages unlabeled programs available online to create a large amount of extra program repair examples, which we use to pre-train our…
• TF-Coder: Program Synthesis for Tensor Manipulations
  19 Mar 2020 2 repositories listed Syntology ran 0 of 14 samples · 14 unverified
  The success and popularity of deep learning is on the rise, partially due to powerful deep learning frameworks such as TensorFlow and PyTorch that make it easier to develop deep learning models.
• Improving Molecular Design by Stochastic Iterative Target Augmentation
  11 Feb 2020 2 repositories listed Syntology ran 0 of 8 samples · 8 unverified
  The property predictor is then used as a likelihood model for filtering candidate structures from the generative model.
• Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations
  5 Oct 2019 2 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  The encoder of TP-N2F employs TPR `binding' to encode natural-language symbolic structure in vector space and the decoder uses TPR `unbinding' to generate, in symbolic space, a sequential program represented by…
• HOUDINI: Lifelong Learning as Program Synthesis
  31 Mar 2018 2 repositories listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  We present a neurosymbolic framework for the lifelong learning of algorithmic tasks that mix perception and procedural reasoning.
• Transductively Informed Inductive Program Synthesis
  20 May 2025 1 repository listed
  Abstraction and reasoning in program synthesis has seen significant progress through both inductive and transductive paradigms.
• PoE-World: Compositional World Modeling with Products of Programmatic Experts
  16 May 2025 1 repository listed
  Learning how the world works is central to building AI agents that can adapt to complex environments.
• Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
  12 May 2025 1 repository listed
  Large Language Models (LLMs) have demonstrated unprecedented capability in code generation.
• Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
  5 May 2025 1 repository listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora.
• OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification
  29 Apr 2025 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  The LLMs are required to understand the provided verification assumption and the potential syntax and semantics space to search for, then generate the complete specification for the potentially buggy operating system…

Syntology lines on 17 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.