Multilingual NLP Benchmarks

Introduction

The field of multilingual NLP has experienced explosive growth, particularly with the rise of LLMs. As models like mBERT, XLM-R, LLaMA, Qwen, and Gemini claim multilingual capabilities, robust evaluation benchmarks become essential for measuring progress and identifying gaps.

This comprehensive guide attempts to index the major multilingual benchmarks available for both discriminative tasks (classification, tagging) and generative tasks (question answering, summarization, translation). Each entry includes paper references, dataset links, and key characteristics.

Why Multilingual Benchmarks Matter

According to recent research analyzing over 2,000 multilingual benchmarks published between 2021-2024, English still dominates evaluation despite significant investments in multilingual evaluation.^[1] Low-resource languages remain severely underrepresented, and there's a notable gap between benchmark performance and real-world human judgments, particularly for traditional NLP tasks.

On a more practical note, having all these benchmarks archived in one place is of great value. I wish I had a resource like this when I first started working on multilingual NLP, it would have saved countless hours of scattered searching. Multilingual evaluation is at the heart of building better multilingual models, and seeing all these benchmarks together offers a chance to reflect on what we have, identify the gaps that remain, and plot a path forward.

Meta-Benchmarks and Evaluation Suites

These comprehensive benchmarks combine multiple tasks to provide holistic evaluation of multilingual models.

XTREME

XTREME-R

XGLUE

MEGA

P-MMEval

Natural Language Understanding Benchmarks

XNLI

PAWS-X

Belebele

Question Answering Benchmarks

MLQA

XQuAD

TyDi QA

MKQA

Machine Translation Benchmarks

FLORES-101 / FLORES-200 / FLORES+

WMT Shared Tasks

Tatoeba

Sequence Labeling Benchmarks

WikiANN / PAN-X

Universal NER (UNER)

Universal Dependencies (UD)

CoNLL Shared Tasks (NER)

Reasoning Benchmarks

XCOPA

MGSM

XL-WiC

Text Classification Benchmarks

SIB-200

Taxi1500

MMS Benchmark

XLM-T

Sentence Retrieval and Similarity

BUCC

STS (Semantic Textual Similarity)

Knowledge and MMLU-Style Benchmarks

MMLU (Original English)

MMMLU (OpenAI)

MMLU-ProX

Global-MMLU

Include

Speech and Multimodal Benchmarks

FLEURS

Fleurs-SLU

Language-Specific Benchmarks

Many languages now have their own dedicated evaluation suites, modeled after GLUE/SuperGLUE.

🇪🇸🇫🇷 Basque - BasqueGLUE

🇧🇬 Bulgarian - bgGLUE

🇩🇪 German - SuperGLEBer

🇭🇺 Hungarian - HuLU

🇮🇹 Italian - UINAUIL

🇰🇷 Korean - KMMLU

🇳🇴 Norwegian - NorBench

🇵🇱 Polish - KLEJ & LEPISZCZE

🇷🇴 Romanian - LiRo

🇷🇺 Russian - MERA

🇸🇪 Swedish - Superlim

🇻🇳 Vietnamese - ViGLUE

🇳🇱 Dutch - DUMB

🇩🇰 Danish - Semantic Reasoning Benchmark

🇪🇸 Catalan - The Catalan Language CLUB

Conclusion

This guide has covered the major multilingual benchmarks available as of late 2025. Several key takeaways are:

1. Coverage is improving but uneven: While benchmarks like FLORES-200 and SIB-200 cover 200+ languages, most benchmarks focus on 10-50 languages, with English, Chinese, Spanish, French, and German being most represented.

2. Task diversity: The field has moved beyond simple classification to include reasoning (XCOPA, MGSM), knowledge evaluation (MMLU variants), and complex QA (TyDi QA).

3. The low-resource gap: Performance on low-resource languages consistently lags behind high-resource languages, highlighting the need for continued investment in underrepresented languages.

Last updated: January 2025

Have a benchmark to add? This list is not exhaustive; the multilingual NLP community continues to create new evaluation resources. Consider contributing to this list through GitHub.

References

1. ^ Wu, M., Wang, W., Liu, S., Yin, H., Wang, X., Zhao, Y., Lyu, C., Wang, L., Luo, W., & Zhang, K. (2025). "The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks." arXiv:2504.15521

Citation

@online{gurgurov2025multilingual,
    title={Multilingual NLP Benchmarks Landscape},
    author={Gurgurov, Daniil},
    year={2026},
    month={January},
    url={https://d-gurgurov.github.io/writing/multilingual-benchmarking.html}
}