Browse State-of-the-Art › Code Repair
Code Repair
14 papers with code · 1 benchmark · 9 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| CodeXGLUE - Bugs2Fix (2 rows) | NSEdit | Fix Bugs with Transformer through a Neural-Symbolic Edit Grammar | — | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
9 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 39 · with a code link: 14 · where Syntology ran a sample: 7 (5 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (7 of 39 tagged: 5 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 39 of 39, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
9 Feb 2021 7 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Benchmark datasets have a significant impact on accelerating research in programming language tasks.
-
14 Aug 2023 3 repositories listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified (of 13 harvested samples)We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on…
-
15 Feb 2023 2 repositories listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 11 unverified (of 19 harvested samples) · 19 pointer-only (licence)Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include…
-
8 Jul 2025 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)On the most challenging tasks, Claude-3 improves from 38.
-
28 Mar 2025 1 repository listedTo address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging.
-
27 Dec 2024 1 repository listedTranslating legacy Fortran code into C++ is a crucial step in modernizing high-performance computing (HPC) applications.
-
19 Sep 2024 1 repository listedDespite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog.
-
COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis9 Aug 2024 1 repository listedIn this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process.
-
18 Jun 2024 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation.
-
23 May 2024 1 repository listedWe introduce AutoCoder, the first Large Language Model to surpass GPT-4 Turbo (April 2024) and GPT-4o in pass@1 on the Human Eval benchmark test (90.
-
16 Nov 2023 1 repository listedINTERVENOR prompts Large Language Models (LLMs) to play distinct roles during the code repair process, functioning as both a Code Learner and a Code Teacher.
-
19 Jun 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it.
-
11 Jun 2021 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)To bridge this gap, we propose a new training approach, Break-It-Fix-It (BIFI), which has two key ideas: (i) we use the critic to check a fixer's output on real bad inputs and add good (fixed) outputs to the training…
-
28 May 2020 1 repository listedAutomated compilation error repair, the problem of suggesting fixes to buggy programs that fail to compile, has generated significant interest in recent years.
-
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks14 Jul 2025 0 repositories listedLarge Language Models (LLMs) have significantly advanced the state-of-the-art in various coding tasks.
-
Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents30 May 2025 0 repositories listedBenchmarks for large language models (LLMs) have predominantly assessed short-horizon, localized reasoning.
-
CrashFixer: A crash resolution agent for the Linux kernel29 Apr 2025 0 repositories listedCode large language models (LLMs) have shown impressive capabilities on a multitude of software engineering tasks.
-
How Accurately Do Large Language Models Understand Code?6 Apr 2025 0 repositories listedLarge Language Models (LLMs) are increasingly used in post-development tasks such as code repair and testing.
-
RocketPPA: Code-Level Power, Performance, and Area Prediction via LLM and Mixture of Experts27 Mar 2025 0 repositories listedThis paper presents RocketPPA, a novel ultra-fast power, performance (delay), and area (PPA) estimator operating directly at the code-level abstraction using HDL code as input.
-
SolBench: A Dataset and Benchmark for Evaluating Functional Correctness in Solidity Code Completion and Repair3 Mar 2025 0 repositories listedSmart contracts are crucial programs on blockchains, and their immutability post-deployment makes functional correctness vital.
-
AuPair: Golden Example Pairs for Code Repair12 Feb 2025 0 repositories listedScaling up inference-time compute has proven to be a valuable strategy in improving the performance of Large Language Models (LLMs) without fine-tuning.
-
LLM-Aided Efficient Hardware Design Automation24 Oct 2024 0 repositories listedWith the rapidly increasing complexity of modern chips, hardware engineers are required to invest more effort in tasks such as circuit design, verification, and physical implementation.
-
Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairs1 Sep 2024 0 repositories listedWith the recent unprecedented advancements in Artificial Intelligence (AI) computing, progress in Large Language Models (LLMs) is accelerating rapidly, presenting challenges in establishing clear guidelines…
-
Is Your AI-Generated Code Really Safe? Evaluating Large Language Models on Secure Code Generation with CodeSecEval2 Jul 2024 0 repositories listedLarge language models (LLMs) have brought significant advancements to code generation and code repair, benefiting both novice and experienced developers.
-
Investigating the Transferability of Code Repair for Low-Resource Programming Languages21 Jun 2024 0 repositories listedTo explain this behavior, we perform a further analysis and find that contrary to preexisting beliefs, the correlation between reasoning ability and code correction ability is weak.
-
Code Repair with LLMs gives an Exploration-Exploitation Tradeoff26 May 2024 0 repositories listedIteratively improving and repairing source code with large language models (LLMs), known as refinement, has emerged as a popular way of generating programs that would be too complex to construct in one shot.
-
DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models19 Feb 2024 0 repositories listedThe automated program repair field has attracted substantial interest over the years, but despite significant research efforts, creating a system that works well for complex semantic bugs such as security…
-
Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Models13 Jan 2024 0 repositories listedWith the recent advancement of Large Language Models (LLMs), generating functionally correct code has become less complicated for a wide array of developers.
-
Enhanced Automated Code Vulnerability Repair using Large Language Models8 Jan 2024 0 repositories listedThis research addresses the complex challenge of automated repair of code vulnerabilities, vital for enhancing digital security in an increasingly technology-driven world.
-
Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation25 Oct 2023 0 repositories listedLarge language models (LLMs) have brought significant advancements to code generation, benefiting both novice and experienced developers.
-
FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair21 Jun 2023 0 repositories listed
-
Poison Attack and Defense on Deep Source Code Processing Models31 Oct 2022 0 repositories listed
-
13 Apr 2022 0 repositories listed
-
Break it - Message it - Fix it : Learning to Repair Python Programs using Error Messages without Labelled Data16 Nov 2021 0 repositories listed
-
Fix Bugs with Transformer through a Neural-Symbolic Edit Grammar16 Nov 2021 0 repositories listed
-
Jointly Learning to Repair Code and Generate Commit Message25 Sep 2021 0 repositories listed
-
Inferring Javascript types using Graph Neural Networks16 May 2019 0 repositories listed
-
Learning to Repair Software Vulnerabilities with Generative Adversarial Networks18 May 2018 0 repositories listed
-
Semantic Code Repair using Neuro-Symbolic Transformation Networks30 Oct 2017 0 repositories listed
Syntology lines on 7 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections