EUROPA Alignment · Technical report

EUROPA Alignment: A Transparent, Parallel Safety Benchmark for 36 European Languages

Birger Moëll

AI Sweden · November 2026 · Version 1.0 · DOI 10.5281/zenodo.24338447

PDF Code & data Answer explorer Zenodo Cite

Abstract

Safety evaluations of large language models are still predominantly English, yet models trained for Europe are expected to behave consistently in two dozen official languages and more. We present EUROPA Alignment, an open, lightweight benchmark and evaluation harness that applies the same 16 original safety scenarios to each of the 36 languages prioritized by OpenEuroLLM (576 items). The scenarios cover refusal calibration, human wellbeing (sycophancy, delusion reinforcement, dependency, manipulation), privacy and prompt injection, high-stakes and fairness-sensitive advice, non-operational dangerous-capability boundaries, and agentic control. Three design choices distinguish it: (i) harmful requests are paired with closely related benign requests so that blanket refusal is penalized; (ii) safety-critical rubric dimensions act as gates whose failure caps the item score, so fluency cannot average away a harmful answer; and (iii) every published run pins target and judge revisions, forbids self-judging, and exposes each answer with its judge rationale.

We apply the benchmark to four checkpoints of a 9B-parameter OpenEuroLLM model across supervised fine-tuning (SFT) and direct preference optimization (DPO). Three findings stand out. First, the choice of SFT mixture dominates: two SFT runs from the same base differ by +0.318 (95% CI +0.282 to +0.352) in mean score, and the weaker one refuses essentially none of the harmful requests. Second, DPO on preference data in only 12 languages raised scores by +0.117 in those languages and by +0.126 in the 23 languages it never saw (p = 0.70): safety preference tuning transferred across languages. Third, aggregate scores hide a language-level failure: every checkpoint scores at most 0.22 on Georgian, where prompts need 6.06× as many tokens as English and every response is truncated. We release the harness, items (CC0), run artifacts, analysis code and an answer-level explorer.

Key results

Cite

If you use EUROPA Alignment or its results, please cite the report:

@techreport{moell2026europaalignmentpaper,
  author      = {Mo{\"e}ll, Birger},
  title       = {{EUROPA} Alignment: A Transparent, Parallel Safety
                 Benchmark for 36 European Languages},
  institution = {AI Sweden},
  type        = {Technical Report},
  year        = {2026},
  month       = nov,
  doi         = {10.5281/zenodo.24338447},
  url         = {https://birgermoell.github.io/europa-alignment/paper/}
}