EUROPA Eval · Technical report

EUROPA Eval: A Parallel, Inspectable Capability Benchmark for Post-Training in 36 European Languages

Birger Moëll

AI Sweden · November 2026 · Version 1.0 · DOI 10.5281/zenodo.24339071

PDF Code & data Answer explorer Zenodo Cite

Abstract

Post-training decides whether a multilingual base model becomes a usable assistant, yet the behaviours post-training is meant to install—answering in the requested format, stopping, grounding answers in context, abstaining when evidence is missing, recovering from tool errors, refusing harmful requests—are rarely measured in parallel across many European languages. We present EUROPA Eval, an open, lightweight benchmark and harness that applies the same 12 task templates to each of the 36 languages prioritized by OpenEuroLLM (432 items), scores 9 templates deterministically under strict output contracts and 3 with a pinned LLM judge, and publishes every answer with its scorer evidence. We evaluate 9 models: five post-training variants of one 9B-parameter OpenEuroLLM base and four open reference models. Four findings stand out. (1) The SFT mixture dominates: two SFT runs from the same base differ by +0.297 (95% CI +0.267 to +0.327). (2) Strict output contracts are a measured capability, not noise: re-scoring the same answers for content alone moves one checkpoint by 0.218 and changes which model ranks first; most of DPO's gain on multiple-choice reading (+0.38) is format compliance (+0.03 on content). (3) DPO on preference data in 12 languages improved the 23 languages it never saw as much as the 12 it did (+0.075 vs +0.083, p = 0.68). (4) Averaging rubric dimensions lets unsafe answers keep credit: 107 responses the judge rated fully unsafe still received a mean item score of 0.300. A ranking from one language agrees with the 36-language ranking only partially (mean Kendall τ = 0.80; English alone τ = 0.79). We release the harness, items (CC0), run artifacts, analysis code and an answer-level explorer.

Key results

Cite

If you use EUROPA Eval or its results, please cite the report:

@techreport{moell2026europaevalpaper,
  author      = {Mo{\"e}ll, Birger},
  title       = {{EUROPA} Eval: A Parallel, Inspectable Capability Benchmark
                 for Post-Training in 36 European Languages},
  institution = {AI Sweden},
  type        = {Technical Report},
  year        = {2026},
  month       = nov,
  doi         = {10.5281/zenodo.24339071},
  url         = {https://birgermoell.github.io/europa-eval/paper/}
}