1.Qwen/Qwen3-Next-80B-A3B-Instruct
exact_match83.2%±0.7%
MGSM
MathReasoningMultilingual
MGSM (Multilingual Grade School Math) is a 250-problem subset of GSM8K hand-translated into ten typologically diverse languages — Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu — yielding 2,500 problem-language pairs. It was built to probe whether chain-of-thought reasoning transfers across languages, including low-resource ones, and the original paper showed that CoT in English on translated problems often outperformed CoT in the source language. The "direct" variant evaluated by this slug uses no chain-of-thought scaffolding and elicits the answer directly, providing a substantially harder lower-bound on multilingual arithmetic capability.
Source paperLatest run: 2026-05-26
Benchmark results
Switch between the canonical ranking, release-date performance view, and score-size tradeoff.
28 models
2.nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
exact_match79.4%±0.7%
3.google/gemma-4-31B-it
exact_match76.6%±0.7%
4.google/Gemma-4-31B-IT-NVFP4
exact_match76.6%±0.7%
5.Qwen/Qwen3.5-122B-A10B-NVFP4
exact_match74.9%±0.7%
6.google/gemma-4-26B-A4B-it
exact_match74.5%±0.7%
7.Qwen/Qwen3.5-35B-A3B
exact_match74.4%±0.7%
8.zai-org/GLM-4.5V-FP8
exact_match73.0%±0.7%
9.Qwen/Qwen3.6-27B
exact_match69.4%±0.7%
10.google/gemma-3-12b-it
exact_match68.4%±0.8%
11.Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
exact_match67.4%±0.8%
12.Qwen/Qwen3.6-35B-A3B
exact_match67.1%±0.8%
13.Qwen/Qwen3-14B
exact_match66.2%±0.8%
14.MiniMax/MiniMax-M2-AWQ
exact_match63.3%±0.8%
15.openai/gpt-oss-120b
exact_match62.2%±0.8%
16.Qwen/Qwen3-32B
exact_match61.1%±0.8%
17.zai-org/GLM-4.5-Air-FP8
exact_match60.8%±0.8%
18.google/gemma-3-27b-it
exact_match60.8%±0.8%
19.openai/gpt-oss-20b
exact_match58.3%±0.8%
20.Qwen/Qwen3-4B
exact_match56.0%±0.8%
21.Qwen/Qwen3-4B-AWQ
exact_match55.5%±0.8%
22.microsoft/phi-4-mini-reasoning
exact_match52.8%±0.8%
23.microsoft/phi-4-mini-instruct
exact_match51.9%±0.8%
24.Qwen/Qwen3-235B-A22B-Thinking-AWQ-2507
exact_match49.3%±0.8%
25.MiniMax/MiniMax-M2.1-AWQ
exact_match48.2%±0.8%
26.Qwen/Qwen3-8B
exact_match47.9%±0.8%
27.microsoft/phi-4
exact_match23.8%±0.7%
28.microsoft/phi-4-reasoning-plus
exact_match23.5%±0.6%
Caveats
Per-language sample size is only 250, so per-language variance is high and small deltas between models on a single language should not be over-interpreted. Translation quality also varies — particularly for low-resource languages such as Swahili, Bengali, and Telugu, where translators may have limited domain vocabulary — so observed gaps reflect translation quality as much as model capability and cannot cleanly be attributed to one factor. Inheritance from GSM8K means MGSM carries forward GSM8K's saturation and contamination concerns: top models exceed 90% on the high-resource splits (Spanish, French, Chinese), and the underlying problem set is publicly available. The mgsm_direct (no-CoT) variant in particular is harder and more sensitive to extraction filtering, so its scores are not comparable to mgsm CoT numbers and the two should never be mixed in a single comparison. Some published evaluations also elicit English chain-of-thought on non-English inputs, conflating language-switching ability with reasoning and muddying interpretation of multilingual capability.
How to cite
Citation
FrozeBench. "MGSM." https://frozebench.com/benchmarks/mgsm-direct. Retrieved 2026-06-04.
BibTeX
@misc{frozebench_mgsm_direct,
title = {MGSM},
howpublished = {\url{https://frozebench.com/benchmarks/mgsm-direct}},
year = {2026},
note = {FrozeBench. Retrieved 2026-06-04.}
}URL
https://frozebench.com/benchmarks/mgsm-direct