OPENEUROLLM · MULTILINGUAL AI SAFETY
Test AI safety
in 36 languages.
Run the same 16 scenarios in every language. Check whether a model can refuse harmful requests, help with legitimate ones, protect private information and follow human instructions. Inspect every answer and judge decision.
v0.1 public development set. Sixteen scenarios per language keep the suite lightweight. Thirty-five generated language packs remain explicitly marked for native review before safety claims or leaderboard use.
ALIGNMENT MATRIX36 × 16
LANGUAGESAFETY BEHAVIOR →
16 scenarios.
36 languages.
36 languages.
See exactly where a model’s behavior changes.
576parallel safety items
36target languages
16scenarios per language
6safety domains
01 / LANGUAGES
36 languages.
The same 16 scenarios.
EUROPA Alignment covers all 36 languages prioritized by OpenEuroLLM. Each one is tested on the same scenarios, with language and script variants recorded in the data. This makes it easier to compare results and find safety failures that appear only in some languages.
Canonical scope: OpenEuroLLM training-data catalogue ↗
02 / SAFETY MAP
Safe is not one behavior.
A model can refuse an obvious attack yet comply in role-play, reinforce a harmful belief, expose personal data or ignore a stop instruction. EUROPA keeps those failure modes separate.
Three response forms
03 / METHOD
Hard boundaries.
Visible evidence.
EUROPA combines strict control-choice checks with a named, revision-pinned LLM judge. Safety-critical dimensions gate the score, so fluent wording cannot average away a dangerous failure.
- 01Pair refusal with assistance
A model must refuse harmful help and still answer closely related benign security questions.
- 02Test the interaction
Single-turn, indirect and real multi-turn scenarios surface failures that simple question answering misses.
- 03Gate critical failures
Privacy, non-discrimination, non-reinforcement and dangerous-capability boundaries cannot be averaged away.
- 04Name the judge
Open responses require an independent pinned judge; every dimension score and concise rationale remains inspectable.
- 05Publish the limits
This suite measures text behavior, not latent capability, deployed tool safety or certification.
04 / RUN
One suite. Your endpoint.
The dependency-free core talks to hosted APIs and Ollama’s native chat endpoint. Transformers is an optional direct backend. Filter one language for diagnostics or run all 576 items for comparable evidence.
export OPENAI_API_KEY="…"
europa-alignment run \
--backend openai-compatible \
--base-url https://api.openai.com/v1 \
--model YOUR_MODEL_ID \
--revision PINNED_VERSION \
--judge-backend openai-compatible \
--judge-model YOUR_JUDGE_ID \
--judge-revision PINNED_JUDGE_VERSION \
--output results/runs/model-api.jsoneuropa-alignment run \
--backend ollama \
--model gemma4:31b \
--revision OLLAMA_MODEL_DIGEST \
--language swe \
--diagnostic \
--output results/runs/gemma-swe.jsonpython3 -m pip install -e '.[local]'
europa-alignment run \
--backend huggingface \
--model /path/to/checkpoint \
--revision CHECKPOINT_SHA \
--judge-backend ollama \
--judge-model YOUR_INDEPENDENT_JUDGE \
--judge-revision JUDGE_DIGEST \
--output results/runs/model-hf.json↻ Resumable after every item⌁ API keys never enter artifacts◇ Raw answers retained
05 / EVIDENCE
Results, with their caveats attached.
The page accepts only complete, non-diagnostic runs with independent pinned target and judge revisions. Open any run to filter by language and inspect the scenario, prior turns, final answer, critical gates and named judge rationale.
No reviewed full-suite runs yet.
The harness is ready. The first contribution should be a reproducible 576-item baseline with independent, pinned target and judge revisions.
SAFETY EVIDENCE EXPLORER
From aggregate score to actual evidence.
Model outputs are untrusted evidence and may contain unsafe text. Private chain-of-thought is not published; prior turns, final answers, critical gates and the named judge’s concise rationale are.
06 / FOUNDATIONS
Grounded in public safety work.
EUROPA Alignment is inspired by contemporary model-card practice and established public safety evaluation research, while contributing a lightweight parallel layer for OpenEuroLLM languages.
35 LANGUAGE PACKS NEED NATIVE REVIEW