Code Repair

Benchmarks archive 2025-07-28

1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

9 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

14 shown of 14 papers with code (39 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
  9 Feb 2021 7 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Benchmark datasets have a significant impact on accelerating research in programming language tasks.
• OctoPack: Instruction Tuning Code Large Language Models
  14 Aug 2023 3 repositories listed Syntology ran 9 of 13 samples · 4 unverified
  We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on…
• Learning Performance-Improving Code Edits
  15 Feb 2023 2 repositories listed Syntology ran 7 of 19 samples · 12 unverified · 19 pointer-only (licence)
  Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include…
• Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving
  8 Jul 2025 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  On the most challenging tasks, Claude-3 improves from 38.
• Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors
  28 Mar 2025 1 repository listed
  To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging.
• Fortran2CPP: Automating Fortran-to-C++ Translation using LLMs via Multi-Turn Dialogue and Dual-Agent Integration
  27 Dec 2024 1 repository listed
  Translating legacy Fortran code into C++ is a crucial step in modernizing high-performance computing (HPC) applications.
• CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair
  19 Sep 2024 1 repository listed
  Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog.
• COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis
  9 Aug 2024 1 repository listed
  In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process.
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
  18 Jun 2024 1 repository listed Syntology ran 5 of 6 samples · 1 unverified
  We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation.
• AutoCoder: Enhancing Code Large Language Model with \textsc{AIEV-Instruct}
  23 May 2024 1 repository listed
  We introduce AutoCoder, the first Large Language Model to surpass GPT-4 Turbo (April 2024) and GPT-4o in pass@1 on the Human Eval benchmark test (90.
• INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair
  16 Nov 2023 1 repository listed
  INTERVENOR prompts Large Language Models (LLMs) to play distinct roles during the code repair process, functioning as both a Code Learner and a Code Teacher.
• Guiding Language Models of Code with Global Context using Monitors
  19 Jun 2023 1 repository listed Syntology ran 2 of 2 samples · 0 unverified
  We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it.
• Break-It-Fix-It: Unsupervised Learning for Program Repair
  11 Jun 2021 1 repository listed Syntology ran 0 of 6 samples · 6 unverified
  To bridge this gap, we propose a new training approach, Break-It-Fix-It (BIFI), which has two key ideas: (i) we use the critic to check a fixer's output on real bad inputs and add good (fixed) outputs to the training…
• MACER: A Modular Framework for Accelerated Compilation Error Repair
  28 May 2020 1 repository listed
  Automated compilation error repair, the problem of suggesting fixes to buggy programs that fail to compile, has generated significant interest in recent years.

Syntology lines on 7 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.