LongBench

LongBench's subtasks emit heterogeneous primary metrics (qa_f1_score, rouge_score, code_sim_score, classification_score, and others), so the benchmark-level primary_metric only resolves cleanly against the QA-style subtask leaves. Runs that target the full LongBench root may surface BENCHMARK_PRIMARY_METRIC_MISSING in this system until the benchmark's aggregate policy is decided, and any LongBench composite score reported in papers averages across incomparable metric scales (F1, ROUGE, exact match, code similarity), so a model can game the headline number by performing well on a single family. Per-subtask reporting is preferable to a single LongBench score whenever discriminating between models matters. The "long-context" framing is also less demanding than it once was. Context lengths in LongBench peak around 30k tokens at the time of publication, well within the context windows of current frontier models, so high LongBench scores no longer indicate strong long- context behavior at the lengths that matter for contemporary retrieval-augmented or agentic workloads. Many subtasks are also retrieval-style with answers concentrated near the start or end of the context, which means positional biases can confound claims of genuine long-range understanding. Coverage is bilingual (English and Chinese only) and per-subtask sample sizes are sometimes only a few hundred examples, amplifying variance. Finally, the few-shot learning subtasks are arguably out of scope for a long-context benchmark — they probe in-context learning more than long-range reading comprehension — so LongBench mixes long-context and in-context-learning signal under one composite.

How to cite

@misc{frozebench_longbench,
  title = {LongBench},
  howpublished = {\url{https://frozebench.com/benchmarks/longbench}},
  year = {2026},
  note = {FrozeBench. Retrieved 2026-06-04.}
}