EUROPA REASONING GitHub

OPENEUROLLM · MULTILINGUAL REASONING

Test reasoning
in 36 languages.

Run the same 24 problems, with the same numbers, in every language and in two modes: direct answers and step-by-step reasoning. See not only whether the answer is right but how the model got there: does its reasoning finish, loop, run out of budget or drift out of the prompt’s language?

Run the reasoning suite How it is scored

v0.1 public development set. Twenty-four problems per language keep the suite lightweight. Thirty-five generated language packs remain explicitly marked for native review before leaderboard use.

REASONING MATRIX36 × 24
LANGUAGEPROBLEM →
24 problems.
36 languages.

See exactly where a model’s reasoning breaks.

864parallel reasoning items
36target languages
24problems per language
4reasoning domains

01 / LANGUAGES

36 languages.
The same 24 problems.

EUROPA Reasoning covers all 36 languages prioritized by OpenEuroLLM. Each one gets the same problems with identical numbers, so a gap between languages is a reasoning gap, not a difficulty gap. Numbers come from a seeded generator, and a fresh seed rebuilds an unseen variant to check for memorization.

Review status is data.English is the authored source pack. Every generated translation carries its model, decoding protocol and review state until a native speaker approves it.

Canonical scope: OpenEuroLLM training-data catalogue ↗

02 / REASONING MAP

Reasoning is not one skill.

A model can do arithmetic yet fall for an intuitive trap, follow a rule yet invent an answer the data does not determine, or reach the right number with an argument that would not convince anyone. EUROPA keeps those apart.

01

Loading reasoning map…

Three response forms

03 / METHOD

Right answers.
Visible process.

EUROPA scores boxed final answers deterministically, judges open explanations with a named, revision-pinned LLM judge, and reports how each answer was reached. An unfinished reasoning trace earns nothing, however promising it looked. Read the short explanation ↓

  1. 01
    Same numbers everywhere

    Seeded parameters give every language the identical problem; gold answers are computed in code, never translated.

  2. 02
    Two modes per model

    Each model runs with thinking off and with thinking on, with a separate reasoning budget, on the same items.

  3. 03
    Score the final answer only

    The last \boxed{} after a closed trace is graded; an unfinished trace is no answer.

  4. 04
    Measure the process

    Trace closure, loops, budget exhaustion, reasoning length and trace script are reported next to accuracy.

  5. 05
    Gate wrong explanations

    An independent pinned judge scores open answers; correctness is a critical dimension that fluent prose cannot average away.

THE SHORT VERSION

What does it test, and how?

The test

Each model answers 24 problems in each of 36 languages: multi-step arithmetic, logic puzzles, classic traps, missing information, base rates and five short explanations, once in default mode and once with reasoning on.

The tooling

The open-source Python harness, adapted from EUROPA Eval and EUROPA Alignment, runs against hosted APIs, Ollama or local Transformers. It keeps the reasoning trace apart from the final answer, saves the pinned model, judge and decoding protocols, and computes process metrics for every item.

The judgment

Nineteen items per language are scored automatically from \boxed{}. Five open explanations go to a separate LLM judge, which scores named criteria from 0–4 and gives a short reason; a failed correctness criterion caps the item score. Published runs use a pinned gemma4:31b judge via Ollama.

This is a public text-only pilot of short, school-level problems, not a definitive leaderboard. Thirty-five language packs still need native review, and automated judgments need human calibration. Read the full methodology

04 / RUN

One suite. Your endpoint.

The dependency-free core talks to hosted APIs and Ollama’s native chat endpoint. Transformers is an optional direct backend. Filter one language for diagnostics or run all 864 items, in both modes, for comparable evidence.

export OPENAI_API_KEY="…"
europa-reasoning run \
  --backend openai-compatible \
  --base-url https://api.openai.com/v1 \
  --model YOUR_MODEL_ID \
  --revision PINNED_VERSION \
  --judge-backend openai-compatible \
  --judge-model YOUR_JUDGE_ID \
  --judge-revision PINNED_JUDGE_VERSION \
  --output results/runs/model-api.json
europa-reasoning run \
  --backend ollama \
  --model gemma4:31b \
  --revision OLLAMA_MODEL_DIGEST \
  --language swe \
  --diagnostic \
  --output results/runs/gemma-swe.json
python3 -m pip install -e '.[local]'
europa-reasoning run \
  --backend huggingface \
  --model /path/to/checkpoint \
  --revision CHECKPOINT_SHA \
  --think --reasoning-tokens 8192 \
  --judge-backend ollama \
  --judge-model YOUR_INDEPENDENT_JUDGE \
  --judge-revision JUDGE_DIGEST \
  --output results/runs/model-hf.json
↻ Resumable after every item⌁ API keys never enter artifacts◇ Raw answers retained

05 / EVIDENCE

Results, with their caveats attached.

The page accepts only complete, non-diagnostic runs with independent pinned target and judge revisions. Open any run to filter by language and inspect the problem, the final answer, its process metrics, critical gates and the named judge’s rationale.

No reviewed full-suite runs yet.

The harness is ready. The first contribution should be a reproducible 864-item baseline with independent, pinned target and judge revisions.

Submit a run

06 / FOUNDATIONS

Grounded in public reasoning research.

EUROPA Reasoning builds on GSM8K, MGSM, GSM-Symbolic and work on overthinking and language mixing in reasoning models, while contributing a lightweight parallel layer for OpenEuroLLM languages.

35 LANGUAGE PACKS NEED NATIVE REVIEW

Help make multilingual reasoning harder to bluff—and easier to inspect.

Review a language packRead the methodology