HumanEval

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

1 dataset whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 135 papers with code (264 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Evaluating Large Language Models Trained on Code
  7 Jul 2021 13 repositories listed Syntology ran 6 of 39 samples · 33 unverified · 2 pointer-only (licence)
  We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities.
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
  25 Mar 2022 8 repositories listed Syntology ran 10 of 10 samples · 0 unverified
  To democratize this, we train and release a family of large language models up to 16.
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
  18 Jun 2024 7 repositories listed Syntology ran 15 of 29 samples · 14 unverified
  We introduce ChatGLM, an evolving family of large language models that we have been developing over time.
• Qwen2 Technical Report
  15 Jul 2024 6 repositories listed
  This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models.
• Reflexion: Language Agents with Verbal Reinforcement Learning
  20 Mar 2023 5 repositories listed Syntology ran 2 of 9 samples · 7 unverified
  Large language models (LLMs) have been increasingly used to interact with external environments (e.
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct
  14 Jun 2023 4 repositories listed Syntology ran 3 of 7 samples · 4 unverified · 1 pointer-only (licence)
  Moreover, our model even outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on HumanEval and HumanEval+.
• StarCoder: may the source be with you!
  9 May 2023 4 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.
• EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization
  24 May 2024 3 repositories listed Syntology ran 12 of 17 samples · 5 unverified
  To address this issue, we propose \textbf{EffiLearner}, a self-optimization framework that utilizes execution overhead profiles to improve the efficiency of LLM-generated code.
• RLHF Workflow: From Reward Modeling to Online RLHF
  13 May 2024 3 repositories listed Syntology ran 0 of 1 samples · 1 unverified · 1 pointer-only (licence)
  We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent…
• Magicoder: Empowering Code Generation with OSS-Instruct
  4 Dec 2023 3 repositories listed Syntology ran 1 of 2 samples · 1 unverified
  Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate diverse instruction data for code.
• OctoPack: Instruction Tuning Code Large Language Models
  14 Aug 2023 3 repositories listed Syntology ran 9 of 13 samples · 4 unverified
  We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on…
• ReCode: Robustness Evaluation of Code Generation Models
  20 Dec 2022 3 repositories listed Syntology ran 5 of 5 samples · 0 unverified
  Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code…
• SelfCodeAlign: Self-Alignment for Code Generation
  31 Oct 2024 2 repositories listed Syntology ran 13 of 37 samples · 24 unverified
  In our primary experiments, we use SelfCodeAlign with CodeQwen1.
• Training Language Models to Self-Correct via Reinforcement Learning
  19 Sep 2024 2 repositories listed
  In particular, we observe that training via SFT falls prey to either a distribution mismatch between mistakes made by the data-collection policy and the model's own responses, or to behavior collapse, where learning…
• Planning In Natural Language Improves LLM Search For Code Generation
  5 Sep 2024 2 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  We empirically demonstrate that this lack of diversity can be mitigated by searching over candidate plans for solving a problem in natural language.
• A Survey on Large Language Models for Code Generation
  1 Jun 2024 2 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of natural language processing (NLP) or software engineering (SE) or both, there is a noticeable absence of a comprehensive…
• MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
  18 May 2024 2 repositories listed Syntology ran 0 of 10 samples · 10 unverified
  In this paper, we introduce a new approach to code generation tasks leveraging multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers.
• CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences
  14 Mar 2024 2 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Evaluating the alignment of large language models (LLMs) with user-defined coding preferences is a challenging endeavour that requires a deep assessment of LLMs' outputs.
• InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
  11 Mar 2024 2 repositories listed Syntology ran 11 of 13 samples · 2 unverified
  We conduct a systematic evaluation for over 100 latest code LLMs on InfiBench, leading to a series of novel and insightful findings.
• Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  19 Feb 2024 2 repositories listed Syntology ran 8 of 11 samples · 3 unverified
  This research aims to accelerate the inference speed of large language models (LLMs) with billions of parameters.
• Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
  13 Feb 2024 2 repositories listed Syntology ran 2 of 4 samples · 2 unverified
  To evaluate code large language models (LLMs), research has relied on a few small manually curated benchmarks, such as HumanEval and MBPP, which represent a narrow part of the real-world software domains.
• Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
  6 Oct 2023 2 repositories listed Syntology ran 7 of 9 samples · 2 unverified
  By leveraging the in-context learning ability of LMs, we integrate Monte Carlo Tree Search into LATS to enable LMs as agents, along with LM-powered value functions and self-reflections for proficient exploration and…
• Baichuan 2: Open Large-scale Language Models
  19 Sep 2023 2 repositories listed
  Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature…
• Code Llama: Open Foundation Models for Code
  24 Aug 2023 2 repositories listed Syntology ran 3 of 5 samples · 2 unverified · 4 pointer-only (licence)
  We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot…
• ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation
  3 Aug 2023 2 repositories listed
  Third, we find that generating the entire class all at once (i.
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation
  13 May 2023 2 repositories listed Syntology ran 3 of 4 samples · 1 unverified
  To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks.
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
  30 Mar 2023 2 repositories listed Syntology ran 2 of 8 samples · 6 unverified
  Large pre-trained code generation models, such as OpenAI Codex, can generate syntax- and function-correct code, making the coding of programmers more productive and our pursuit of artificial general intelligence closer.
• Multi-lingual Evaluation of Code Generation Models
  26 Oct 2022 2 repositories listed Syntology ran 1 of 6 samples · 5 unverified
  Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of…
• Rethinking Verification for LLM Code Generation: From Generation to Testing
  9 Jul 2025 1 repository listed
  Large language models (LLMs) have recently achieved notable success in code-generation benchmarks such as HumanEval and LiveCodeBench.
• any4: Learned 4-bit Numeric Representation for LLMs
  7 Jul 2025 1 repository listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  We present any4, a learned 4-bit weight quantization solution for large language models (LLMs) providing arbitrary numeric representations without requiring pre-processing of weights or activations.

Syntology lines on 25 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.