Datasets › MBPP

MBPP (Mostly Basic Python Programming)

Introduced by Jacob Austin et al. in Program Synthesis with Large Language Models16 Aug 2021 archive 2025-07-28

The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry-level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases.

Benchmarks archive 2025-07-28

All 1 leaderboard whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 57 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 4 placed): Code Generation · MBPP (57 judged, 4 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Code Generation MBPP EG-CFG (DeepSeek-V3-0324) Accuracy 96.6 Execution Guided Line-by-Line Code Generation boazlavon/eg_cfg 99 Compare

Papers archive 2025-07-28

papers with a benchmark row: 34 · with a code link: 30 · where Syntology ran a sample: 21 (20 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (21 of 34 with a benchmark row: 20 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 34 of 34 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 666. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
Execution Guided Line-by-Line Code Generation 1 2 12 Jun 2025 not harvested
CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging 1 1 8 Feb 2025 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks 0 1 20 Jan 2025 not harvested
Planning-Driven Programming: A Large Language Model Programming Workflow 1 1 21 Nov 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified
AFlow: Automating Agentic Workflow Generation 4 1 14 Oct 2024 not harvested
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging 1 2 2 Oct 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified
MapCoder: Multi-Agent Code Generation for Competitive Problem Solving 2 3 18 May 2024 community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified
SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents 0 1 23 Mar 2024 not harvested
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM 1 2 12 Mar 2024 not harvested
The Claude 3 Model Family: Opus, Sonnet, Haiku 0 3 4 Mar 2024 not harvested
StarCoder 2 and The Stack v2: The Next Generation 4 1 29 Feb 2024 not harvested
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence 1 8 25 Jan 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence)
Mixtral of Experts 6 1 8 Jan 2024 5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks 2 2 5 Jan 2024 not harvested
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation 1 2 20 Dec 2023 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair 1 3 16 Nov 2023 not harvested
Mistral 7B 6 1 10 Oct 2023 official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models 2 2 6 Oct 2023 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Textbooks Are All You Need II: phi-1.5 technical report 1 1 11 Sep 2023 not harvested
Code Llama: Open Foundation Models for Code 2 14 24 Aug 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (4 pointer-only for licence)
How Does Naming Affect LLMs on Code Analysis Tasks? 0 5 24 Jul 2023 not harvested
Llama 2: Open Foundation and Fine-Tuned Chat Models 19 4 18 Jul 2023 community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified (20 pointer-only for licence)
WizardCoder: Empowering Code Large Language Models with Evol-Instruct 4 1 14 Jun 2023 community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
PaLM 2 Technical Report 1 1 17 May 2023 not harvested
StarCoder: may the source be with you! 4 3 9 May 2023 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Teaching Large Language Models to Self-Debug 2 7 11 Apr 2023 0 ran · 1 unverified
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X 2 1 30 Mar 2023 official (archive's flag): 2 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (1 pointer-only for licence)
LLaMA: Open and Efficient Foundation Language Models 57 4 27 Feb 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified (4 pointer-only for licence)
LEVER: Learning to Verify Language-to-Code Generation with Execution 1 1 16 Feb 2023 official (archive's flag): 18 ran · 18 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 2 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 4 unverified (1 pointer-only for licence)
Coder Reviewer Reranking for Code Generation 1 10 29 Nov 2022 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
CodeT: Code Generation with Generated Tests 1 5 21 Jul 2022 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified
Natural Language to Code Translation with Execution 1 1 25 Apr 2022 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 3 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (2 pointer-only for licence)
InCoder: A Generative Model for Code Infilling and Synthesis 3 1 12 Apr 2022 official (archive's flag): 2 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified (2 pointer-only for licence)
PaLM: Scaling Language Modeling with Pathways 7 2 5 Apr 2022 32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified

Dataset loaders archive 2025-07-28

No loader listed in the archive.

Tasks archive 2025-07-28

License archive 2025-07-28

No licence recorded in the archive. Absence here is not a statement about the dataset's terms.

Modalities archive 2025-07-28

No modality tagged.

Languages archive 2025-07-28

No language tagged.

Variants archive 2025-07-28

  • MBPP

1 variant name, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections