Code Completion

Benchmarks archive 2025-07-28

6 leaderboard tables shown for this task, 6 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

12 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 108 papers with code (212 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
  9 Feb 2021 7 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Benchmark datasets have a significant impact on accelerating research in programming language tasks.
• StarCoder 2 and The Stack v2: The Next Generation
  29 Feb 2024 4 repositories listed
  Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size.
• DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioning
  2 Sep 2024 3 repositories listed
  Through extensive experimental analysis, we identified three key challenges in designing effective data management strategies that enable the model to achieve long-context capability without sacrificing performance in…
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
  10 Oct 2023 3 repositories listed
  In long context scenarios, large language models (LLMs) face three main challenges: higher computational cost, performance reduction, and position bias.
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  28 Aug 2023 3 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding.
• Open Vocabulary Learning on Source Code with a Graph-Structured Cache
  18 Oct 2018 3 repositories listed
  Machine learning models that take computer program source code as input typically use Natural Language Processing (NLP) techniques.
• On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards
  4 Jul 2024 2 repositories listed
  Foundation models (FM), such as large language models (LLMs), which are large-scale machine learning (ML) models, have demonstrated remarkable adaptability in various downstream software engineering (SE) tasks, such as…
• Optimizing Large Language Models for OpenAPI Code Completion
  24 May 2024 2 repositories listed
  This study evaluates the OpenAPI completion performance of GitHub Copilot, a prevalent commercial code completion tool, and proposes a set of task-specific optimizations leveraging Meta's open-source model Code Llama.
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion
  12 Mar 2024 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse.
• Scope is all you need: Transforming LLMs for HPC Code
  18 Aug 2023 2 repositories listed
  With easier access to powerful compute resources, there is a growing trend in the field of AI for software development to develop larger and larger language models (LLMs) to address a variety of programming tasks.
• MPI-rical: Data-Driven MPI Distributed Parallelism Assistance with Transformers
  16 May 2023 2 repositories listed
  Message Passing Interface (MPI) plays a crucial role in distributed memory parallelization across multiple nodes.
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation
  13 May 2023 2 repositories listed Syntology ran 3 of 4 samples · 1 unverified
  To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks.
• CodeKGC: Code Language Model for Generative Knowledge Graph Construction
  18 Apr 2023 2 repositories listed Syntology ran 1 of 16 samples · 15 unverified
  However, large generative language model trained on structured data such as code has demonstrated impressive capability in understanding natural language for structural prediction and reasoning tasks.
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
  23 Feb 2023 2 repositories listed Syntology ran 0 of 2 samples · 2 unverified
  Large Language Models (LLMs) are increasingly being integrated into various applications.
• Multi-lingual Evaluation of Code Generation Models
  26 Oct 2022 2 repositories listed Syntology ran 1 of 6 samples · 5 unverified
  Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of…
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation
  8 Mar 2022 2 repositories listed Syntology ran 2 of 4 samples · 2 unverified
  Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task.
• Energy-Based Models for Code Generation under Compilability Constraints
  9 Jun 2021 2 repositories listed
  Neural language models can be successfully trained on source code, leading to applications such as code completion.
• Neural Software Analysis
  16 Nov 2020 2 repositories listed
  The resulting tools complement and outperform traditional program analyses, and are used in industrial practice.
• Structural Language Models of Code
  30 Sep 2019 2 repositories listed Syntology ran 0 of 8 samples · 8 unverified
  We introduce a new approach to any-code completion that leverages the strict syntax of programming languages to model a code snippet as a tree - structural language modeling (SLM).
• Seed-Coder: Let the Code Model Curate Data for Itself
  4 Jun 2025 1 repository listed
  Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs.
• SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development
  22 May 2025 1 repository listed Syntology ran 5 of 5 samples · 0 unverified · 5 pointer-only (licence)
  We therefore introduce SWE-Dev, the first large-scale dataset (with 14, 000 training and 500 test samples) designed to evaluate and train autonomous coding systems on real-world feature development tasks.
• Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective
  15 May 2025 1 repository listed
  In this paper, we first propose CoV-Eval, a multi-task benchmark covering various tasks such as code completion, vulnerability repair, vulnerability detection and classification, for comprehensive evaluation of LLM code…
• ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding
  27 Mar 2025 1 repository listed
  Language models (LMs) have become a staple of the code-writing toolbox.
• LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation
  25 Mar 2025 1 repository listed
  We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV Cache in large language model (LLM) inference, delivering substantial memory savings while preserving superior performance.
• CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection
  12 Mar 2025 1 repository listed
  LLMs perform exceptionally well in the CASTLE dataset when identifying vulnerabilities in small code snippets.
• Enhancing High-Quality Code Generation in Large Language Models with Comparative Prefix-Tuning
  12 Mar 2025 1 repository listed
  Unlike existing methods that require training multiple prefixes, our approach trains only one prefix and leverages pairs of high-quality and low-quality code samples, introducing a sequence-level ranking loss to guide…
• LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification
  24 Feb 2025 1 repository listed Syntology ran 2 of 10 samples · 8 unverified
  Despite its promise, the effective application of speculative decoding in LLMs still confronts three key challenges: the increasing memory demands of the draft model, the distribution shift between the short-training…
• How to Get Your LLM to Generate Challenging Problems for Evaluation
  20 Feb 2025 1 repository listed
  The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60% accuracy, thereby demonstrating the effectiveness of our framework at generating challenging problems.
• GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generation
  19 Jan 2025 1 repository listed
  Results show that our method reduces the energy consumption between 23-50 % on average for code generation tasks without significantly affecting accuracy.
• GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models
  5 Nov 2024 1 repository listed
  The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent version updates while maintaining compatibility with previous versions.

Syntology lines on 11 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.