Program Repair

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

4 leaderboard tables shown for this task, 4 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

9 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

6 subtasks in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 52 papers with code (132 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• AutoCodeRover: Autonomous Program Improvement
  8 Apr 2024 5 repositories listed Syntology ran 0 of 4 samples · 4 unverified · 1 pointer-only (licence)
  Recent progress in Large Language Models (LLMs) has significantly impacted the development process, where developers can use LLM-based programming assistants to achieve automated coding.
• Agentless: Demystifying LLM-based Software Engineering Agents
  1 Jul 2024 4 repositories listed Syntology ran 3 of 24 samples · 21 unverified
  However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents?
• xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
  6 Mar 2023 4 repositories listed Syntology ran 1 of 6 samples · 5 unverified
  Recently, pre-trained large language models (LLMs) have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving…
• C-Pack of IPAs: A C90 Program Benchmark of Introductory Programming Assignments
  17 Jun 2022 3 repositories listed
  Due to the vast number of students enrolled in Massive Open Online Courses (MOOCs), there has been an increasing number of automated program repair techniques focused on introductory programming assignments (IPAs).
• RepairBench: Leaderboard of Frontier Models for Program Repair
  27 Sep 2024 2 repositories listed
  The key characteristics of RepairBench are: 1) it is execution-based: all patches are compiled and executed against a test suite, 2) it assesses frontier models in a frequent and standardized way.
• HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale
  9 Sep 2024 2 repositories listed
  Large Language Models (LLMs) have revolutionized software engineering (SE), showcasing remarkable proficiency in various coding tasks.
• Teaching Large Language Models to Self-Debug
  11 Apr 2023 2 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.
• Exploring Plausible Patches Using Source Code Embeddings in JavaScript
  31 Mar 2021 2 repositories listed
  Despite the immense popularity of the Automated Program Repair (APR) field, the question of patch validation is still open.
• Unified Pre-training for Program Understanding and Generation
  10 Mar 2021 2 repositories listed
  Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models.
• Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
  20 May 2020 2 repositories listed Syntology ran 4 of 6 samples · 2 unverified
  Second, we present a self-supervised learning paradigm for program repair that leverages unlabeled programs available online to create a large amount of extra program repair examples, which we use to pre-train our…
• Learning and Evaluating Contextual Embedding of Source Code
  21 Dec 2019 2 repositories listed
  We fine-tune CuBERT on our benchmark tasks, and compare the resulting models to different variants of Word2Vec token embeddings, BiLSTM and Transformer models, as well as published state-of-the-art models, showing that…
• Neural Program Repair by Jointly Learning to Localize and Repair
  3 Apr 2019 2 repositories listed Syntology ran 0 of 8 samples · 8 unverified
  We show that it is beneficial to train a model that jointly and directly localizes and repairs variable-misuse bugs.
• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
  24 Dec 2018 2 repositories listed
  This paper presents a novel end-to-end approach to program repair based on sequence-to-sequence learning.
• CoSIL: Software Issue Localization via LLM-Driven Code Repository Graph Searching
  28 Mar 2025 1 repository listed
  CoSIL reduces the search space through module call graphs, iteratively searches the function call graph to obtain relevant contexts, and uses context pruning to control the search direction and manage contexts…
• Less is More: Adaptive Program Repair with Bug Localization and Preference Learning
  9 Mar 2025 1 repository listed
  Automated Program Repair (APR) is a task to automatically generate patches for the buggy code.
• o3-mini vs DeepSeek-R1: Which One is Safer?
  30 Jan 2025 1 repository listed
  The irruption of DeepSeek-R1 constitutes a turning point for the AI industry in general and the LLMs in particular.
• Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repair
  5 Dec 2024 1 repository listed
  We propose DEVLoRe to use issue content (description and message) and stack error traces to localize buggy methods, then rely on debug information in buggy methods and issue content and stack error to localize buggy…
• Planning-Driven Programming: A Large Language Model Programming Workflow
  21 Nov 2024 1 repository listed Syntology ran 0 of 9 samples · 9 unverified
  LPW also sets new state-of-the-art Pass@1 accuracy, achieving 98.
• From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging
  2 Oct 2024 1 repository listed Syntology ran 8 of 9 samples · 1 unverified
  While large language models have made significant strides in code generation, the pass rate of the generated code is bottlenecked on subtle errors, often requiring human intervention to pass tests, especially for…
• Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs
  16 Sep 2024 1 repository listed
  LLMs have long demonstrated remarkable effectiveness in automatic program repair (APR), with OpenAI's ChatGPT being one of the most widely used models in this domain.
• RePair: Automated Program Repair with Process-based Feedback
  21 Aug 2024 1 repository listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  The gap between the trepidation of program reliability and the expense of repairs underscores the indispensability of Automated Program Repair (APR).
• SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning
  3 Jun 2024 1 repository listed Syntology ran 11 of 15 samples · 4 unverified
  This approach led to the development of SemCoder, a Code LLM with only 6.
• Benchmarking Educational Program Repair
  8 May 2024 1 repository listed Syntology ran 12 of 13 samples · 1 unverified · 13 pointer-only (licence)
  The emergence of large language models (LLMs) has sparked enormous interest due to their potential application across a range of educational tasks.
• Aligning the Objective of LLM-based Program Repair
  13 Apr 2024 1 repository listed Syntology ran 6 of 8 samples · 2 unverified
  In this paper, we investigate a new approach to adapt LLMs to program repair.
• RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
  25 Mar 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Unlike existing deep learning-based approaches, which prompt a model with a fixed prompt or in a fixed feedback loop, our work treats the LLM as an agent capable of autonomously planning and executing actions to fix…
• Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing
  19 Feb 2024 1 repository listed
  In this paper, we propose shifting the focus of robustness evaluation for Neural Program Repair (NPR) techniques toward naturally-occurring data transformations.
• A Novel Approach for Automatic Program Repair using Round-Trip Translation with Large Language Models
  15 Jan 2024 1 repository listed
  We investigate whether this correction capability of Large Language Models (LLMs) extends to Automatic Program Repair (APR).
• RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
  25 Dec 2023 1 repository listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  This results in RepairLLaMA producing a highly effective `program repair adapter' for fixing bugs with AI.
• Breaking the Silence: the Threats of Using LLMs in Software Engineering
  13 Dec 2023 1 repository listed
  Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code…
• Out of Context: How important is Local Context in Neural Program Repair?
  8 Dec 2023 1 repository listed
  Our results indicate that overall repair success increases with the size of the local context (albeit not for all bug types) and confirm the common practice that roughly 50-60% of the input window should be used for…

Syntology lines on 14 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.