MBPP (Mostly Basic Python Programming)
Introduced by Jacob Austin et al. in
Program Synthesis with Large Language Models 16 Aug 2021 archive 2025-07-28
The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry-level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases.
Benchmarks archive 2025-07-28
All 1 leaderboard whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.
Syntology 57 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 4 placed): Code Generation · MBPP (57 judged, 4 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
Papers archive 2025-07-28
papers with a benchmark row: 34 · with a code link: 30 · where Syntology ran a sample: 21 (20 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show:
all papers with a benchmark row only where code ran (21 of 34 with a benchmark row: 20 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 34 of 34 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).
The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 666. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed 2026-10-05 ; when this build read Syntology's graph is in the build record .
Paper Code Results Date Samples run Syntology
Execution Guided Line-by-Line Code Generation
1
2
12 Jun 2025
not harvested
CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging
1
1
8 Feb 2025
3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks
0
1
20 Jan 2025
not harvested
Planning-Driven Programming: A Large Language Model Programming Workflow
1
1
21 Nov 2024
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified
AFlow: Automating Agentic Workflow Generation
4
1
14 Oct 2024
not harvested
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging
1
2
2 Oct 2024
official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified
MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
2
3
18 May 2024
community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified
SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents
0
1
23 Mar 2024
not harvested
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
1
2
12 Mar 2024
not harvested
The Claude 3 Model Family: Opus, Sonnet, Haiku
0
3
4 Mar 2024
not harvested
StarCoder 2 and The Stack v2: The Next Generation
4
1
29 Feb 2024
not harvested
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
1
8
25 Jan 2024
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence )
Mixtral of Experts
6
1
8 Jan 2024
5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
2
2
5 Jan 2024
not harvested
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
1
2
20 Dec 2023
official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence )
INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair
1
3
16 Nov 2023
not harvested
Mistral 7B
6
1
10 Oct 2023
official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence )
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
2
2
6 Oct 2023
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence )
Textbooks Are All You Need II: phi-1.5 technical report
1
1
11 Sep 2023
not harvested
Code Llama: Open Foundation Models for Code
2
14
24 Aug 2023
official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (4 pointer-only for licence )
How Does Naming Affect LLMs on Code Analysis Tasks?
0
5
24 Jul 2023
not harvested
Llama 2: Open Foundation and Fine-Tuned Chat Models
19
4
18 Jul 2023
community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified (20 pointer-only for licence )
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
4
1
14 Jun 2023
community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence )
PaLM 2 Technical Report
1
1
17 May 2023
not harvested
StarCoder: may the source be with you!
4
3
9 May 2023
2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Teaching Large Language Models to Self-Debug
2
7
11 Apr 2023
0 ran · 1 unverified
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
2
1
30 Mar 2023
official (archive's flag): 2 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (1 pointer-only for licence )
LLaMA: Open and Efficient Foundation Language Models
57
4
27 Feb 2023
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified (4 pointer-only for licence )
LEVER: Learning to Verify Language-to-Code Generation with Execution
1
1
16 Feb 2023
official (archive's flag): 18 ran · 18 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 2 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 4 unverified (1 pointer-only for licence )
Coder Reviewer Reranking for Code Generation
1
10
29 Nov 2022
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence )
CodeT: Code Generation with Generated Tests
1
5
21 Jul 2022
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified
Natural Language to Code Translation with Execution
1
1
25 Apr 2022
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 3 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (2 pointer-only for licence )
InCoder: A Generative Model for Code Infilling and Synthesis
3
1
12 Apr 2022
official (archive's flag): 2 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified (2 pointer-only for licence )
PaLM: Scaling Language Modeling with Pathways
7
2
5 Apr 2022
32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified
Dataset loaders archive 2025-07-28
No loader listed in the archive.
Tasks archive 2025-07-28
License archive 2025-07-28
No licence recorded in the archive. Absence here is not a statement about the dataset's terms.
Modalities archive 2025-07-28
No modality tagged.
Languages archive 2025-07-28
No language tagged.
Variants archive 2025-07-28
1 variant name, as the archive lists them.