Code Generation

Description from the archive archive 2025-07-28; Papers-with-Code links inside it are rewritten to this site.

Benchmarks archive 2025-07-28

27 leaderboard tables shown for this task, 27 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 27 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

70 datasets whose archive record lists this task, ordered by the archive's paper count. 30 shown of 70 until expanded.

Subtasks archive 2025-07-28

5 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 745 papers with code (1,697 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• LLaMA: Open and Efficient Foundation Language Models
  27 Feb 2023 57 repositories listed Syntology ran 26 of 58 samples · 32 unverified · 4 pointer-only (licence)
  We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters.
• Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks
  11 Apr 2016 42 repositories listed Syntology ran 1 of 9 samples · 8 unverified · 3 pointer-only (licence)
  Face detection and alignment in unconstrained environment are challenging due to various poses, illuminations and occlusions.
• Llama 2: Open Foundation and Fine-Tuned Chat Models
  18 Jul 2023 19 repositories listed Syntology ran 31 of 52 samples · 21 unverified · 16 pointer-only (licence)
  In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
• Evaluating Large Language Models Trained on Code
  7 Jul 2021 13 repositories listed Syntology ran 6 of 39 samples · 33 unverified · 2 pointer-only (licence)
  We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities.
• GPT-4 Technical Report
  15 Mar 2023 11 repositories listed Syntology ran 2 of 5 samples · 3 unverified · 1 pointer-only (licence)
  We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs.
• pix2code: Generating Code from a Graphical User Interface Screenshot
  22 May 2017 11 repositories listed
  Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications.
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
  10 Oct 2023 8 repositories listed Syntology ran 0 of 5 samples · 5 unverified
  We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models.
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
  25 Mar 2022 8 repositories listed Syntology ran 10 of 10 samples · 0 unverified
  To democratize this, we train and release a family of large language models up to 16.
• SantaCoder: don't reach for the stars!
  9 Jan 2023 7 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code.
• PaLM: Scaling Language Modeling with Pathways
  5 Apr 2022 7 repositories listed Syntology ran 30 of 37 samples · 7 unverified
  To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
  9 Feb 2021 7 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Benchmark datasets have a significant impact on accelerating research in programming language tasks.
• StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing
  20 Jun 2018 7 repositories listed
  Semantic parsing is the task of transducing natural language (NL) utterances into formal meaning representations (MRs), commonly represented as tree structures.
• Mixtral of Experts
  8 Jan 2024 6 repositories listed Syntology ran 5 of 5 samples · 0 unverified
  In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
• Mistral 7B
  10 Oct 2023 6 repositories listed Syntology ran 9 of 11 samples · 2 unverified · 1 pointer-only (licence)
  We introduce Mistral 7B v0.
• A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
  7 Jul 2017 6 repositories listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  Automated documentation of programming source code and automated code generation from natural language are challenging tasks of both practical and scientific interest.
• A Syntactic Neural Model for General-Purpose Code Generation
  6 Apr 2017 6 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  We consider the problem of parsing natural language descriptions into source code written in a general-purpose programming language like Python.
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
  2 Sep 2021 5 repositories listed Syntology ran 1 of 11 samples · 10 unverified
  We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers.
• Content Enhanced BERT-based Text-to-SQL Generation
  16 Oct 2019 5 repositories listed
  We present a simple methods to leverage the table content for the BERT-based model to solve the text-to-SQL problem.
• Qwen3 Technical Report
  14 May 2025 4 repositories listed
  In this work, we present Qwen3, the latest version of the Qwen model family.
• AFlow: Automating Agentic Workflow Generation
  14 Oct 2024 4 repositories listed
  Large language models (LLMs) have demonstrated remarkable potential in solving complex tasks across diverse domains, typically by employing agentic workflows that follow detailed instructions and operational sequences.
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
  22 Jun 2024 4 repositories listed Syntology ran 4 of 6 samples · 2 unverified
  In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions.
• StarCoder 2 and The Stack v2: The Next Generation
  29 Feb 2024 4 repositories listed
  Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size.
• How is ChatGPT's behavior changing over time?
  18 Jul 2023 4 repositories listed Syntology ran 2 of 6 samples · 4 unverified · 5 pointer-only (licence)
  We find that the performance and behavior of both GPT-3.
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct
  14 Jun 2023 4 repositories listed Syntology ran 3 of 7 samples · 4 unverified · 1 pointer-only (licence)
  Moreover, our model even outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on HumanEval and HumanEval+.
• StarCoder: may the source be with you!
  9 May 2023 4 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  12 Apr 2022 4 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants.
• Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
  20 Aug 2021 4 repositories listed
  The most notable of these comes in the form of the first self-described `AI pair programmer', GitHub Copilot, a language model trained over open-source GitHub code.
• TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation
  5 Oct 2018 4 repositories listed Syntology ran 2 of 7 samples · 5 unverified · 2 pointer-only (licence)
  We present TRANX, a transition-based neural semantic parser that maps natural language (NL) utterances into formal meaning representations (MRs).
• ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation
  24 May 2025 3 repositories listed Syntology ran 3 of 3 samples · 0 unverified
  We showcase the utility of this dataset through: 1) improving infographic chart understanding via fine-tuning, 2) benchmarking code generation for infographic charts, and 3) enabling example-based infographic chart…
• EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization
  24 May 2024 3 repositories listed Syntology ran 12 of 17 samples · 5 unverified
  To address this issue, we propose \textbf{EffiLearner}, a self-optimization framework that utilizes execution overhead profiles to improve the efficiency of LLM-generated code.

Syntology lines on 23 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.