OpenEuroLLM 9B post-training plan: instruct → reasoning → tools → DPO → RL

Status 2026-10-23 (evening). This is the master plan: what the post-training pipeline is for, every decision taken and why, what is done, what runs now, and what comes next, with budgets and gates. It is written so that the whole pipeline can be reproduced and continued by someone new.

Document Role
this file the plan: goals, decisions, stage plans, budget, gates, open questions
OELLM_POST_TRAINING_PLAYBOOK.md the record: exact data, configs, hashes, job IDs, costs, results, pitfalls, reproduction commands
OELLM_RL_RUN_PROGRAM.md Stage 5 run by run: questions, run cards, results, stop rules, merging policy
OELLM_STAGE5_RL_PLAN.md Stage 5 data pools, rewards, difficulty windows, environment tiers
oellm-rlvr repo RL control plane (configs, Slurm rendering, verifier patches, data scripts)

1. Goal and principles

Turn the OpenEuroLLM 9B 256K base (openeurollm/oellm-9b-256k-theta64m-prelude-anneal300b) into an assistant that answers well in the 36 OpenEuroLLM languages, has an opt-in reasoning mode that finishes its traces, calls tools reliably, follows format instructions, is safe, and is measurably better at verifiable tasks after RL.

Principles that every stage follows:

  1. Selected vs candidate: a stage starts from the previous stage’s selected checkpoint; a candidate is promoted only if its paired evaluations pass the gates (§8). Otherwise the next stage starts from the previous selection.
  2. One tokenizer, our templates: tokenizer tokenizer.json sha256 5123f74b…, templates v2 (Stages 1–2) and v2.1 (Stage 3 on) pinned by sha256 in every data manifest and asserted by every launcher. Never another run’s tokenizer or template (the threeway DPO release ships a broken template).
  3. Pre-tokenize offline, reject never truncate, label guards in every trainer (no supervised <think>, tool responses or tool lists), manifests with per-source counts.
  4. Every job above ~20 GPU-h gets a local-h estimate before submission; one question per experiment.
  5. Both modes stay trained: default mode must never emit think tags; thinking mode must close its trace.
  6. Models private until evaluation and license review; datasets and benchmarks public.

2. Status dashboard

Stage Status Checkpoint / job Cost (measured) Key result
1 Instruct SFT ✅ done birgermoell/oellm-9b-256k-stage1-instruct-v1 58e42ab4 3,276 EUROPA Eval 0.600, Alignment 0.455
2 Reasoning SFT ✅ done …-stage2-reasoning-v1 2db72459 2,811 Eval 0.609, Align 0.605; thinking mode loops (53 % unfinished)
3 Tool-use SFT ✅ done …-stage3-tooluse-v1 v1.1 88875629 (tag-row repair) 541 Eval 0.616, Align 0.642; tool JSON 99.5 %, decision 0.87
4 DPO ⚠️ v2 trained, gate failed on tools v2: 250K-pair subset, 1,954 steps (job 62116464) → oellm-dpo/output/stage4-dpo-v2-250k-bf16; Stage-3 ↔ v2 merges α 0.5/0.75 under evaluation ~4.4K val loss 0.315, pref. acc 0.85; reasoning ≈ Stage 3; tool JSON 0.995 → 0.898 (runaway calls)
5 RL 🔬 ablations RL-0, RL-0b, A2a, A1, A3, A4, A5 done; P1 (A3 recipe from DPO v2) and A4b (multi-turn tool RL from DPO v2) queued ~1.9K thinking accuracy 0.435 → 0.53, unfinished 49 % → 34 %

Budget: ~13.6K of ~50K local-h spent since 2026-10-20 (10.2K measured 10-24 + DPO v2 ~3.2K + evals; scripts/leonardo_budget.py).

3. Decisions log (dated, with reasons)

Date Decision Why
10-20 Two-stage SFT with a hard enable_thinking switch (template v2); <think> always prompt-side the model never learns to open a think block; default mode stays clean
10-21 Stage 2 mix by token budget, 25 % direct replay keep default mode while adding reasoning
10-21 Stage 3 = tool SFT with ~30 % “do not call” rows, translated slice in 16 languages; template v2.1 When2Call lesson: over-calling; tool sources are English
10-22 Repair collapsed lm_head rows of tag tokens 11–18 → Stage-3 v1.1 (v1 never emitted </tool_call>) low-LR SFT cannot separate rows that start equal; repair is cheap and verified
10-22 Stage 4 = replicate the threeway DPO run (1.44M pairs) with our tokenizer/template, full size reference run lifted EUROPA Eval 0.762 → 0.840
10-22 Add multilingual format-compliance DPO pairs EUROPA Eval gap traced to strict-format SFT targets (99 % ignore “answer only with …”)
10-23 DPO at 2 pairs per GPU per micro-step 1 pair/GPU was FSDP all-gather bound (25.3 s/step ⇒ ~20K local-h)
10-23 RL ablations on Stage-3 v1.1 in parallel with DPO; RL program with gates and stop rules RL work does not wait for DPO; one question per run
10-23 Merging policy: checkpoint averaging; Stage-3→DPO interpolation as DPO fallback; never with v1 cheap insurance against DPO hurting thinking/tool use
10-23 Keep the non-stop penalty (−0.5 for unfinished traces) in RL A2a: fewest unfinished traces, best MATH-500
10-23 Reasoning-language policy pending A1: answers must be in the user’s language (hard rule); the trace language is decided by A1’s accuracy cost RL-0 switched EU-language traces to English (24.8 % → 1.1 % in prompt language)
10-23 Reasoning language: traces in any language, final answer in the prompt language (multilingual_math check: answer in W1–W3); native reasoning later as an opt-in A1’s trace-language reward cost 11.6 points on native EU math, concentrated in smaller languages with unfinished traces; DeepSeek-R1 reported the same kind of cost
10-23 Code track: EvalPlus baseline + single-turn code RL (A5) with a contained execution server only ~7K code-reasoning SFT rows; code is verifiable
10-23 oellm-eval (project harness, v0.3.0) at stage boundaries; comparison set: our base, Neonkraft SFT, threeway DPO, Apertus-8B-Instruct, Qwen3-8B project comparability and knowledge-retention checks

4. Stages 1–3 (done) — summary

Full details, data tables and reproduction commands are in the playbook §5–7.

Known weaknesses entering Stage 4: format compliance in default mode (EUROPA numeric 0.06, 122 malformed answers), thinking-mode termination, reasoning traces mostly English, Basque and Georgian weakest on knowledge benchmarks.

5. Stage 4 — DPO (in progress)

Recipe (replication of birgermoell/oellm-9b-threeway-dpo-full-v1, only tokenizer/template changed): TRL 0.29 DPO, sigmoid loss, β 0.1, LR 5e-7 cosine, 3 % warmup, global batch 128, max length 2,048 (longer pairs dropped at build time), 1 epoch, seed 42, bf16 + FSDP full shard, gradient checkpointing; policy and reference = Stage-3 v1.1.

Data (built 10-23, job 61856512): HelpSteer3 preference @ f6d14577 (ties dropped), Dolci-Instruct-DPO-translated @ a57841a6 (12 languages, every 100th row held out), UltraFeedback-cleaned @ 770076f0 → 1,441,722 train / 15,827 validation pairs (reference 1,441,547 / 15,824). Pre-tokenized with our tokenizer and template v2.1 in default mode; pairs containing tag IDs 11–18 dropped.

Trainer: scripts/dpo_train_pretok.py (TRL subclass that trains on the pre-tokenized ids; manifest, template and tokenizer hash checks; vectorized shard scan; per-batch tag guard), config configs/dpo_oellm_stage4_v1_leonardo.yaml, launcher slurm/dpo_oellm_stage4_v1_leonardo.sbatch (self-queued continuation jobs, resume from the latest checkpoint, smoke mode with a resume test).

Run so far: 2-node smoke passed (incl. resume). Full run at 1 pair/GPU reached step 500 (checkpoint saved, 136 GB) at 25.3 s/step — all-gather bound, ~79 h / ~20K local-h for the epoch — and was stopped. It resumes from step 500 at 2 pairs/GPU, grad-accum 2 (same global batch and recipe), job 61922663, queued since 10-23 morning (≈10K jobs pending on the partition). Health at step ~400: loss 0.43, preference accuracy 0.80, margin 1.2. Estimate: 35–45 h on 8 nodes, ~9–11K local-h for the full epoch (to be replaced by the measured speed after the first 30 min of the resumed job). The user approved the full set (10-23).

Update 10-24/25 — DPO v2 instead of the full set. The resumed 4-node run at 2 pairs/GPU ran out of memory at step 1 (four times, the chain repeated the crash; ~106 local-h; chains now stop after a crash). The full epoch at the measured 25.3 s/step would cost ~19K local-h, so v2 trains one epoch on a stratified 249,996-pair subset (scripts/build_oellm_dpo_subset.py, config configs/dpo_oellm_stage4_v2_250k_leonardo.yaml): all 14,769 format-compliance pairs (the planned follow-up format pass folded in), all 28,935 HelpSteer3, 30,000 UltraFeedback, 14,691 Dolci-translated per language (12). 1 pair/GPU on 4 nodes, 44 s/step, 23 h, ~3.2K local-h. Validation loss 0.369 → 0.315, preference accuracy 0.85, margin 2.2. Gate result: quick reasoning eval ≈ Stage 3 (thinking 0.440 vs 0.435, unfinished 47 % vs 49 %, default 0.558 vs 0.581); default-mode replies ~2.6× longer (generation check median 39 → 115 tokens). Held-out tool check regressed: decision 0.870 → 0.831, names 0.734 → 0.654, JSON 0.995 → 0.898, arguments 0.981 → 0.849, finished 98 % → 90 % — mostly runaway generation inside tool calls (30 of 36 unfinished stop mid call, repeated near-identical calls or looping argument text), not tag errors. Cause: tool-free, default-mode pairs plus DPO’s length push. Merge result (10-25, jobs 62268191–62268202):

Model default acc thinking acc / unfinished tool decision names JSON args finished
Stage-3 v1.1 0.581 0.435 / 49 % 0.870 0.734 0.995 0.981 98 %
DPO v2 (α 1.0) 0.558 0.440 / 47 % 0.831 0.654 0.898 0.849 90 %
merge α 0.75 0.533 0.442 / 47 % 0.839 0.679 0.920 0.862 91 %
merge α 0.5 0.568 0.426 / 47 % 0.878 0.767 0.985 0.951 98 %

α 0.5 restores tool use (decision and names above Stage 3; JSON/args slightly below) at equal reasoning; it is the Stage-4 candidate pending EUROPA Eval/Alignment (does it keep DPO’s chat-quality gain?). oellm-dpo/output/stage4-dpo-v2-250k-merge-a0.5.

Next (as planned before the merge result): (1) Stage-3 ↔ v2 linear merges at α 0.5 / 0.75 (slurm/merge_interpolate_leonardo.sbatch) with tool and reasoning evals; (2) P1 and A4b RL runs from v2 (RL program); (3) EUROPA Eval/Alignment on the selected candidate (private HF upload); (4) if neither merge nor A4b restores tools, DPO v3 with tool-call and thinking-mode pairs.

Stage 4b — judge-labelled on-policy DPO for refusal calibration and grounding (planned after W2, user decision 10-26). EUROPA showed regressions after DPO v2 + merge: grounding & safety 0.70 → 0.58 (A4b; 0.62 at W1a) and refusal calibration 0.44 → 0.35. Pipeline (scripts/build_oellm_safety_dpo_prompts.py, scripts/judge_onpolicy_dpo.py, slurm/judge_onpolicy_dpo_leonardo.sbatch):

Still to do in Stage 4 (original list):

  1. Format-compliance pairs (builder scripts/build_oellm_format_pairs.py, launcher slurm/build_oellm_format_pairs_leonardo.sbatch; first build 10-23 night: 18,062 prompts — 11,500 math final-answer-only in 23 languages, 6,562 letter-only MCQ in 10 languages from exams_qa + slovak_financial_exam — Stage-3 v1.1 answered in the bare requested form for 12 of 6,562 MCQ prompts and 0 of 11,500 math prompts — every math reply is a worked solution). Built: 14,769 train / 295 validation pairs (8,628 math, 6,141 MCQ; 2,985 prompts dropped because the reply did not finish within 512 tokens), pre-tokenized with the main DPO guards in $SCRATCH/oellm-dpo/format-v1/pretok (median 208 tokens). Plan: a short follow-up DPO pass on the Stage-4 checkpoint with these pairs mixed 1:2 with a sample of the main pairs (same recipe), then EUROPA Eval strict-format items as the check. Because the model never samples bare math answers, RL alone cannot teach this (A3 canary: math_direct reward 0), so this pass must precede the RL windows. Design: strict-format prompts (number, letter, one word, JSON-only, N lines, no explanation) in the 36 languages, EUROPA items excluded; chosen = verified bare answer, rejected = Stage-3 v1.1’s own verbose/wrong answer (vLLM, ~2–4 GPU-h). They go into a short follow-up DPO pass on the Stage-4 checkpoint (or the next DPO version), since the main run is already training.
  2. Gates (§8) on the final export; also the tag-row check and the thinking-mode/tool checks, because all DPO pairs are default-mode and tool-free.
  3. Merging fallback: average of the last 2–3 checkpoints vs the best single; if thinking or tools regress, interpolate Stage-3 v1.1 → DPO at α ∈ {0.5, 0.75, 1.0} before falling back to Stage 3.
  4. Export BF16 with the v2.1 template, upload private, copy off $SCRATCH (40-day purge).

6. Stage 5 — RL

Run-by-run detail in OELLM_RL_RUN_PROGRAM.md; pools and rewards in OELLM_STAGE5_RL_PLAN.md.

Stack: oellm-rlvr + TMAX (open-instruct fork 3f80d370), DAPO, β 0, 2 nodes per run (4 ZeRO-3 learners with CPU-offloaded DeepSpeedCPUAdam, 4 vLLM engines); qualified on Leonardo after six canaries (fixes in playbook §10). Measured cost: ~2.2 local-h per 128 rollouts at ~3.8K tokens; a 48-update ablation ≈ 50–80 local-h.

Ablation results so far (quick reasoning eval, held out, thinking mode):

Run Change Accuracy Unfinished MATH-500 EU math Trace in prompt language
Stage-3 v1.1 — 0.435 48.9 % 0.160 0.108 24.8 %
RL-0 math DAPO, unclosed ⇒ 0 0.512 39.3 % 0.140 0.308 1.1 %
RL-0b RL-0 + 48 updates 0.532 37.4 % 0.180 0.367 0.4 %
A2a non-stop penalty −0.5 0.523 34.0 % 0.210 0.300 1.5 %

A1 (language-consistency reward on the EU-language rows): native traces kept (65 % on native EU math vs 2.5 % for RL-0), overall accuracy equal (0.514), but EU-math accuracy 0.192 vs 0.308 for RL-0 — native traces finish less often (worst in cs/hu/nl/sv/bg/fi); not adopted. Decided: traces in any language, answer-language check (RL program, A1). Default-mode accuracy unchanged in all (0.575–0.593 vs 0.581). Running: A5 (single-turn code RL, 2,004 oellm-code-rlvr problems, contained execution server). Still to build: A3 (default-mode strict-format/numeric prompts in RL, per-row mode switch), A4 (single-turn tool reward).

Production sequence (from the selected Stage-4 checkpoint): W1 warm-up (4K responses, math + exam-MCQ + strict format + light IF) → W2 core (8K, + harder math, code, IF, tool calls) → W3 frontier (16K, hardest pools) → T1/T2 tool RL (single-turn, then multi-turn with SkyRL) → F final selection. Each window: difficulty profiling on the current checkpoint (pass@8 bins), realized-mix logging, checkpoints every 16 updates, gates before promotion, stop rules (entropy collapse, length blow-up, > 25 % at length cap, < 20 % mixed groups, default-mode think tags, language drift, trace legibility).

Reasoning-language policy (decided 2026-10-23 after A1): the final answer is in the user’s language, enforced in W1–W3 by the multilingual_math answer-language check (check: answer). Traces may be in any language (in practice English) but must stay legible natural language (legibility monitored: detector confidence, glued-word share, perplexity under the Stage-3 model). Native-language reasoning is revisited as an opt-in after SFT on good native traces, starting with the languages where A1 kept accuracy (de, fr, es, it, pl), and informed by the 16K re-eval of A1’s unfinished native items (job 62021559).

7. Code track

8. Evaluation protocol and gates

Harness What When Cost
EUROPA Eval (432 items, 36 languages) + Alignment (576) capabilities incl. strict format; safety/over-refusal every stage ~2 h, 1 node
Quick reasoning eval (570 items, both modes) accuracy, unfinished rate, length, per language every RL run, every stage ~1.4 h, 1 node
Trace-language audit (lingua) share of traces in the prompt language every RL run CPU
EUROPA Reasoning (864 items, 36 languages) reasoning accuracy + process metrics (closure, loops, budget, trace script) stage boundaries, RL windows ~1 node-h per mode
Held-out tool check (361 rows) call decision, JSON validity, argument validity, think leaks Stage 3+, after DPO, tool RL short
Code: EvalPlus (HumanEval+ 164, MBPP+ 378) pass@1 base/plus, both modes baseline, after code RL, stage boundaries ~1 node-h
oellm-eval v0.3.0 (Belebele, Global-MMLU, INCLUDE, MultiBLiMP, Global-MGSM) knowledge/language retention, project comparability stage boundaries ~180 local-h per model
Tag-row check teacher-forced top-1 of tag tokens 11–18 after every training stage minutes

Gates (all stages): no think tags in default-mode answers; identity correct; EUROPA Eval ≥ previous stage; Alignment not regressed (Stage 4: ≥ 0.85); oellm-eval within noise of the previous stage; stage-specific checks (tool parse rate, trace closure, malformed answers ≤ 40 after DPO; RL windows: target holdouts up on a paired test, no retention regression, closure targets). Selection always on frozen holdouts, never on training reward.

Reference comparisons: public oellm-eval scores (openeurollm/prelude-base-eval-scores) for Prelude, datamix, Apertus-8B, Olmo-3-7B, Llama-3.1-8B, Qwen3-8B-Base, Qwen3.5-9B-Base (playbook §8.1). Stage-3 v1.1 leads all European and open base models in the file on Belebele/Global-MMLU/INCLUDE and trails Qwen3/3.5 by 5–9 points. Own runs (10-23, array 61949325; playbook §8.1): versus our own base, post-training added +8 (Belebele), +4 (Global-MMLU), +12 (INCLUDE) points with grammar unchanged; Stage 3 matches the Neonkraft SFT and the threeway DPO, beats Apertus-8B-Instruct on reading/Global-MMLU but not on INCLUDE, and trails Qwen3-8B by 4–8 points on knowledge.

9. Budget

Block Spent (measured) Remaining plan (estimate)
Stages 1–3 (SFT, translation, data, exports) ~7.1K —
Stage 4 DPO 1.0K ~9–11K (full epoch at the new speed) + format-pair pass ~0.3K
Stage 5 RL ablations 0.9K ~0.3K (A1, A5, A3/A4)
Stage 5 production (W1–W3, T1/T2, final) — ~12–14K
Evaluations (all harnesses, comparison models) ~0.3K ~2K
Total 9.34K ~24–28K → ~34–37K of the 50K working budget

Every job above ~20 GPU-h gets an estimate first; the DPO estimate is replaced by the measured speed once the resumed job runs.

10. Risks and open questions

Risk / question Mitigation / decision point
DPO queue wait (8 nodes on a full partition) keep queued; if it waits > 24 h, consider 4 nodes (better per-GPU efficiency, longer wall time)
DPO weakens thinking mode or tool use (all pairs default-mode, tool-free) post-DPO gates; Stage-3 → DPO interpolation; T1 tool RL
RL drifts traces to English / less legible text A1 decision; language and legibility stop rules; answer-language hard rule
Small, easy RL pools saturate (RL-0b flattening) W1–W3 use larger, harder pools with measured difficulty
Code RL signal too sparse A5 mixed-group rate; SFT top-up if needed
Strict-format habit relearned by a future SFT rewrite/drop the 9.4K non-compliant targets in the next instruct-data version
oellm-eval container gaps (PolyMath, Macedonian INCLUDE) run PolyMath from our own venv; report upstream
$SCRATCH 40-day purge copy selected exports to work storage and HF (private)

11. Next steps (in order)

  1. ✅ Language policy decided (traces any language, answers in the prompt language). Collect A5 (code) and the 16K A1 re-eval.
  2. Collect the oellm-eval comparison (5 models) and add it to the playbook table.
  3. DPO resume starts → record measured speed and cost → run to the end with self-queued continuations.
  4. Build the format-compliance pairs while DPO trains; build A3 (default-mode RL rows) and A4 (tool reward).
  5. Post-DPO: export, gates, merging checks, select the Stage-4 checkpoint.
  6. Profile W1 pools on the Stage-4 checkpoint → W1 → W2 → W3, each gated; T1 inside W2 if A4 passes.
  7. SkyRL qualification on Leonardo for T2 (multi-turn tools with error recovery, multilingual Wikipedia search).
  8. Final selection, release evaluation (fresh-seed EUROPA Reasoning variant), model card, private upload.

12. Release hygiene (all stages)

Pinned source revisions, per-row source and license, deterministic selection, rejection reports, pre-tokenized data equal to the audited tokens, private model uploads until evaluation and license review, one template file per stage with tests proving existing data renders identically, HF token only via stdin, no personal paths or account names in public repositories.