EUROPA Alignment · Technical report
EUROPA Alignment: A Transparent, Parallel Safety Benchmark for 36 European Languages
Abstract
Safety evaluations of large language models are still predominantly English, yet models trained for Europe are expected to behave consistently in two dozen official languages and more. We present EUROPA Alignment, an open, lightweight benchmark and evaluation harness that applies the same 16 original safety scenarios to each of the 36 languages prioritized by OpenEuroLLM (576 items). The scenarios cover refusal calibration, human wellbeing (sycophancy, delusion reinforcement, dependency, manipulation), privacy and prompt injection, high-stakes and fairness-sensitive advice, non-operational dangerous-capability boundaries, and agentic control. Three design choices distinguish it: (i) harmful requests are paired with closely related benign requests so that blanket refusal is penalized; (ii) safety-critical rubric dimensions act as gates whose failure caps the item score, so fluency cannot average away a harmful answer; and (iii) every published run pins target and judge revisions, forbids self-judging, and exposes each answer with its judge rationale.
We apply the benchmark to four checkpoints of a 9B-parameter OpenEuroLLM model across supervised fine-tuning (SFT) and direct preference optimization (DPO). Three findings stand out. First, the choice of SFT mixture dominates: two SFT runs from the same base differ by +0.318 (95% CI +0.282 to +0.352) in mean score, and the weaker one refuses essentially none of the harmful requests. Second, DPO on preference data in only 12 languages raised scores by +0.117 in those languages and by +0.126 in the 23 languages it never saw (p = 0.70): safety preference tuning transferred across languages. Third, aggregate scores hide a language-level failure: every checkpoint scores at most 0.22 on Georgian, where prompts need 6.06× as many tokens as English and every response is truncated. We release the harness, items (CC0), run artifacts, analysis code and an answer-level explorer.
Key results
- Mean score: Stage-1 SFT 0.455 → Dolci SFT 0.773 → DPO final 0.893.
- Harmful/benign phishing pair calibrated in 0 of 36 languages for Stage-1 SFT, 24 of 36 after DPO; 8 languages still over-refuse the benign request after DPO.
- DPO gains are as large in the 23 languages absent from the preference data as in the 12 present.
- Georgian is the weakest language for every checkpoint because of tokenizer fertility and truncation, not missing safety training.
Cite
If you use EUROPA Alignment or its results, please cite the report:
@techreport{moell2026europaalignmentpaper,
author = {Mo{\"e}ll, Birger},
title = {{EUROPA} Alignment: A Transparent, Parallel Safety
Benchmark for 36 European Languages},
institution = {AI Sweden},
type = {Technical Report},
year = {2026},
month = nov,
doi = {10.5281/zenodo.24338447},
url = {https://birgermoell.github.io/europa-alignment/paper/}
}