EUROPA EVAL GitHub

OPENEUROLLM · 36-LANGUAGE EVALUATION

36 languages.
One honest protocol.

EUROPA Eval asks the same practical questions in every OpenEuroLLM language—then keeps language, task, format failures and judge evidence visible.

Run the pilot Explore the repository

v0.1 translation-review pilot. The harness is production-shaped; 35 generated language packs are clearly marked for native review before leaderboard use.

CAPABILITY ATLAS36 · v0.1
LANGUAGECAPABILITY →
Same task.
Every language.

Averages are useful. Language-level failures are the finding.

432parallel pilot items
36target languages
12tasks per language
5capability domains

01 / LANGUAGES

The whole target set.
Nothing hidden in “multilingual.”

The registry follows OpenEuroLLM’s current 36 macro-languages and preserves its internal language/script variants. Every language receives exactly the same template coverage.

Review status is data.English is the authored source pack. Every generated translation carries its model, decoding protocol and review state until a native speaker approves it.

Canonical scope: OpenEuroLLM training-data catalogue ↗

02 / CAPABILITIES

A score should leave a shape.

A model can sound fluent in one language and still miss dates, invent an absent fact or break a JSON contract in another. EUROPA keeps those failures separate.

01

Loading capability map…

Six task forms

03 / METHOD

Parallel by design.
Inspectable by default.

EUROPA uses deterministic scoring wherever possible and makes every result carry its language, protocol and judge provenance.

  1. 01
    Hold templates constant

    Twelve semantic tasks repeat across all 36 language packs.

  2. 02
    Profile, don’t flatten

    Language, domain, task, coverage, weakest slice and uncertainty stay visible.

  3. 03
    Prefer deterministic checks

    Exact, numeric, JSON and format contracts run locally without a judge.

  4. 04
    Name the judge

    Open answers remain unscored until a pinned judge is configured.

  5. 05
    Separate protocols

    Reasoning on and off are distinct runs; model and judge revisions travel with every answer.

04 / RUN

One suite. Your endpoint.

The dependency-free core talks to hosted APIs and Ollama’s native chat endpoint. Transformers is an optional direct backend. Filter one language for diagnostics or run all 432 items for comparable evidence.

export OPENAI_API_KEY="…"
europa run \
  --backend openai-compatible \
  --base-url https://api.openai.com/v1 \
  --model YOUR_MODEL_ID \
  --revision PINNED_VERSION \
  --judge-backend openai-compatible \
  --judge-model YOUR_JUDGE_ID \
  --output results/runs/model-api.json
europa run \
  --backend ollama \
  --model gemma4:31b \
  --revision OLLAMA_MODEL_DIGEST \
  --language swe \
  --diagnostic \
  --output results/runs/gemma-swe.json
python3 -m pip install -e '.[local]'
europa run \
  --backend huggingface \
  --model /path/to/checkpoint \
  --revision CHECKPOINT_SHA \
  --output results/runs/model-hf.json
↻ Resumable after every item⌁ API keys never enter artifacts◇ Raw answers retained

05 / EVIDENCE

Results, with their caveats attached.

The page accepts only complete, non-diagnostic runs. Reasoning-on and reasoning-off remain separate protocols. Open any run to filter by language and inspect every question, final answer, scoring decision and named judge rationale.

No reviewed full-suite runs yet.

The harness is ready. The first contribution should be a reproducible 432-item baseline with pinned target and judge revisions.

Submit a run

06 / EXTERNAL CHECK

How does this compare
with EuroEval?

EUROPA Eval is a small, zero-shot probe of post-training capabilities across the 36 OpenEuroLLM languages. EuroEval tests a broader set of European-language tasks and reports a within-leaderboard rank score. The two scores measure different things; neither is a percentage translation of the other.

8EUROPA model runs checked
1same model ID found
0weight revisions confirmed
Public European all-models leaderboard cross-check · 12 October 2026
EUROPA runEUROPA pilotEuroEval evidence
Apertus-8B-Instruct-250957.3%Same model ID; scores for 13 languages, but no European aggregate or confirmed weight revision.
gemma4:31b77.3%Related google/gemma-4-31B-it validation entry: rank score 1.65 ± 0.12. Ollama weights and protocol not verified as equivalent.
gemma4:e2b64.6%Related google/gemma-4-E2B-it validation entry: rank score 2.91 ± 0.14. Not an exact checkpoint match.
qwen3.6:35b-a3b88.1%Different-generation Qwen3.5-35B-A3B-FP8 validation entry: rank score 1.63 ± 0.12. Context only, not a checkpoint match.
Other four runs—No same-ID entry found in the European table; excluded from comparison.

EuroEval rank scores are lower-is-better and depend on the models and tasks in that leaderboard. EUROPA percentages are item scores from a public, mostly machine-translated pilot. Do not subtract these columns.

WHAT THE EXISTING RUNS SUGGEST

31 / 36matching pairwise orderings

In the 12 languages shared by Apertus and the two Gemma entries, their relative order agrees in 31 of 36 model-pair checks. Gemma 31B leads both alternatives in every shared language; Apertus and Gemma E2B reverse order in five. For example, EUROPA favors E2B in Estonian (56.8% vs 54.7%), while EuroEval favors Apertus (rank score 3.19 vs 3.79). These are model-family comparisons, not identical checkpoints or independent observations. Qwen3.6 is excluded because EuroEval's Qwen3.5 entry is a different model generation.

07 / ECOSYSTEM

Built with the ecosystem.

EUROPA adds a small, inspectable cross-language capability layer. It complements deeper native datasets and OpenEuroLLM’s established evaluation infrastructure.

35 LANGUAGE PACKS NEED NATIVE REVIEW

Help make European model evaluation harder to fool—and easier to trust.

Review a language packRead the methodology