Text-to-Code Generation

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

10 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

12 shown of 12 papers with code (20 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
  9 Feb 2021 7 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Benchmark datasets have a significant impact on accelerating research in programming language tasks.
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
  2 Sep 2021 5 repositories listed Syntology ran 1 of 11 samples · 10 unverified
  We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers.
• Magicoder: Empowering Code Generation with OSS-Instruct
  4 Dec 2023 3 repositories listed Syntology ran 1 of 2 samples · 1 unverified
  Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate diverse instruction data for code.
• Bridging the Gap Between Open-Source and Proprietary LLMs in Table QA
  11 Jun 2025 1 repository listed
  This paper presents a system developed for SemEval 2025 Task 8: Question Answering (QA) over tabular data.
• Planning-Driven Programming: A Large Language Model Programming Workflow
  21 Nov 2024 1 repository listed Syntology ran 0 of 9 samples · 9 unverified
  LPW also sets new state-of-the-art Pass@1 accuracy, achieving 98.
• InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct
  8 Jul 2024 1 repository listed Syntology ran 11 of 11 samples · 0 unverified · 11 pointer-only (licence)
  Recent advancements in open-source code large language models (LLMs) have been driven by fine-tuning on the data generated from powerful closed-source LLMs, which are expensive to obtain.
• Can Large Language Models Solve Robot Routing?
  16 Mar 2024 1 repository listed
  We systematically investigate the performance of LLMs in robot routing by constructing a dataset with 80 unique robot routing problems across 8 variants in both single and multi-robot settings.
• Guiding Language Models of Code with Global Context using Monitors
  19 Jun 2023 1 repository listed Syntology ran 2 of 2 samples · 0 unverified
  We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it.
• Code Execution with Pre-trained Language Models
  8 May 2023 1 repository listed
  Code execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code.
• C3PO: A Lightweight Copying Mechanism for Translating Pseudocode to Code
  20 Nov 2022 1 repository listed
  In the Copy Phase, a binary classifier is employed to determine and mask the pseudocode tokens that can be directly copied into the code.
• PanGu-Coder: Program Synthesis with Function-Level Language Modeling
  22 Jul 2022 1 repository listed
  We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i.
• StructCoder: Structure-Aware Transformer for Code Generation
  10 Jun 2022 1 repository listed Syntology ran 1 of 8 samples · 7 unverified
  This paper addresses the problem of code generation, where the goal is to generate target code given source code in a different language or a natural language description.

Syntology lines on 7 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.