1.nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
pass@194.0%±1.1%
MBPP
Coding
MBPP (Mostly Basic Python Problems) is a dataset of 974 crowd-sourced entry-level Python programming tasks, each pairing a short natural-language description with a reference solution and three unit tests that automatically score model output. Problems target programmers with up to a year of Python experience and cover basic data structures, string manipulation, and simple algorithms. MBPP and its sanitized 427-problem subset became one of the standard probes for code-generation capability in pre-trained language models, and remain widely reported despite the field having moved toward more discriminative coding benchmarks.
Source paperLatest run: 2026-05-25
Benchmark results
Switch between the canonical ranking, release-date performance view, and score-size tradeoff.
28 models
2.Qwen/Qwen3-Next-80B-A3B-Instruct
pass@179.6%±1.8%
3.Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
pass@178.6%±1.8%
4.microsoft/phi-4
pass@172.2%±2.0%
5.Qwen/Qwen3-14B
pass@171.4%±2.0%
6.google/gemma-3-27b-it
pass@171.4%±2.0%
7.Qwen/Qwen3-235B-A22B-Thinking-AWQ-2507
pass@165.8%±2.1%
8.Qwen/Qwen3-8B
pass@165.6%±2.1%
9.google/gemma-3-12b-it
pass@165.4%±2.1%
10.MiniMax/MiniMax-M2.1-AWQ
pass@160.4%±2.2%
11.Qwen/Qwen3-4B-AWQ
pass@158.0%±2.2%
12.Qwen/Qwen3.5-35B-A3B
pass@155.2%±2.2%
13.microsoft/phi-4-mini-instruct
pass@154.8%±2.2%
14.Qwen/Qwen3.5-122B-A10B-NVFP4
pass@148.6%±2.2%
15.Qwen/Qwen3-32B
pass@147.4%±2.2%
16.Qwen/Qwen3.6-27B
pass@126.8%±2.0%
17.google/gemma-4-26B-A4B-it
pass@124.4%±1.9%
18.openai/gpt-oss-120b
pass@19.8%±1.3%
19.google/Gemma-4-31B-IT-NVFP4
pass@17.2%±1.2%
20.Qwen/Qwen3.6-35B-A3B
pass@13.4%±0.8%
21.MiniMax/MiniMax-M2-AWQ
pass@11.2%±0.5%
22.microsoft/phi-4-reasoning-plus
pass@10.4%±0.3%
23.microsoft/phi-4-mini-reasoning
pass@10.2%±0.2%
24.Qwen/Qwen3-4B
pass@10.2%±0.2%
25.openai/gpt-oss-20b
pass@10.0%±0.0%
26.zai-org/GLM-4.5-Air-FP8
pass@10.0%±0.0%
27.google/gemma-4-31B-it
pass@10.0%±0.0%
28.zai-org/GLM-4.5V-FP8
pass@10.0%±0.0%
Caveats
The most consequential limitation is unit-test coverage: with only three tests per problem, both false positives (incorrect solutions passing all three tests) and false negatives (correct solutions failing edge cases that were never tested) are common. The EvalPlus family of benchmarks was created specifically to address this by augmenting MBPP with auto-generated stress tests, and observed score drops on MBPP+ relative to vanilla MBPP confirm that headline pass@1 numbers overstate true correctness rates by several points. MBPP is also saturated for frontier models, with pass@1 scores routinely above 80-90%, leaving little discriminative headroom. The dataset has been public since 2021 and is widely contaminated in modern training corpora, so high scores partially reflect memorization. Coverage is restricted to Python, so MBPP says nothing about multi-language coding ability or about more realistic software- engineering scenarios involving multiple files, dependencies, or iterative debugging. Crowd-sourced problem statements are also of uneven quality, with some descriptions ambiguous enough that "correct" behavior is open to interpretation.
How to cite
Citation
FrozeBench. "MBPP." https://frozebench.com/benchmarks/mbpp. Retrieved 2026-06-04.
BibTeX
@misc{frozebench_mbpp,
title = {MBPP},
howpublished = {\url{https://frozebench.com/benchmarks/mbpp}},
year = {2026},
note = {FrozeBench. Retrieved 2026-06-04.}
}URL
https://frozebench.com/benchmarks/mbpp