Stage 5 — RL run program

Status 2026-10-25: RL-0, RL-0b, A2a, A1, A3, A4, A5 done (results below); Stage-4 DPO v2 trained, evals running. This is the run-by-run plan for Stage 5: which RL runs we make, in what order, what each one must answer, what it costs, and how its result changes the next run. The data pools, rewards and environment tiers are specified in OELLM_STAGE5_RL_PLAN.md; results go into the playbook §10 as each run finishes.

1. What RL must deliver

The target is a 9B model that is clearly better than Stage 4 at things we can verify, in all 36 languages, without losing what SFT and DPO built. Concretely, by the end of Stage 5:

Capability Today (Stage-3 v1.1) Stage-5 target Measured by
Thinking traces that finish 51 % closed at 6K (quick eval); 81 % of unclosed traces stay unclosed at 16K, half are loops ≥ 95 % closed within the window budget, loops ≈ 0 quick reasoning eval, EUROPA Reasoning process metrics
Reasoning accuracy (thinking mode) 0.435 (quick eval) clear paired gain on frozen holdouts MATH-500, AIME, MGSM, PolyMath, EUROPA Reasoning
Reasoning in the user’s language 24.8 % of non-English traces in the prompt language (RL-0: 1.1 %) answers always in the user’s language; traces per A1 (≥ 90 % native if the cost is ≤ 2 points, otherwise legible English) lingua audit on quick-eval traces; EUROPA Reasoning trace-script metric
Direct-mode answers and strict formats EUROPA numeric 0.056, 122 malformed numeric ≥ 0.5, malformed ≤ 20 EUROPA Eval
Tool use decision 0.870, JSON 0.995; EUROPA tool-recovery 0.486 decision ≥ 0.92, recovery ≥ 0.8 held-out tool eval, EUROPA Eval, BFCL
Retention EUROPA Eval 0.616 / Alignment 0.642 (before DPO) no regression vs the Stage-4 checkpoint EUROPA Eval/Alignment, identity, default-mode think-tag rate = 0

2. Budget envelope

Working budget 50K local-h (account total 400K, shared). Used by 2026-10-23 evening: 9.34K (sacct), of which RL

  1. Planned DPO: ~9–11K for the full epoch (revised after the speed measurement). RL envelope: ~25K local-h with a ~5K reserve, so the program can absorb one failed window or one extra ablation.

Cost model (measured, canary 6, job 61859252, 2 nodes): 16 prompts × 8 samples, mean response 3.8K tokens, max 6,144: ~125 s per update ≈ 2.2 local-h per 128 rollouts. Cost scales roughly with generated tokens, so

local-h per update ≈ 2.2 × (rollouts / 128) × (mean response tokens / 3,800)

Estimates below use this formula; each run re-measures it and the playbook records the real number.

3. Principles every run follows

  1. One question per run. Ablations change one thing against a fixed baseline; production windows change only what the ablations decided.
  2. A run card before submission (template in §9): parent checkpoint, hypothesis, data with the realized mix, reward, settings, budget, live stop rules, end-of-run evaluation, decision rule.
  3. Select on frozen holdouts, never on training reward. Checkpoints every 16 updates; the selected one is the best on the target holdout among those that pass retention.
  4. Profile difficulty on the current checkpoint (pass@8 bins, Stage-5 plan §4) before each window, and check the realized mix per batch.
  5. Check the special-token rows after every run: the teacher-forced top-1 measurement that repair_tag_head_rows.py records as before (a measure-only flag is still to be added). RL updates are tiny, so the rows should not collapse again, but we confirm the closers still win at closing positions.
  6. Both modes stay trained. Every production window includes default-mode prompts (no think tokens allowed); default-mode think-tag rate must stay 0.
  7. Ablations run on Stage-3 v1.1 while DPO trains, so RL work does not wait for DPO. Their conclusions are relative (A vs B on the same parent) and are re-checked once on the DPO checkpoint in W1.

4. The run sequence

 RL-0 think probe ──► A-series ablations (parallel with DPO) ──► Stage-4 DPO checkpoint
                                                               │
                         W1 warm-up (4K) ◄─────────────────────┘
                              │ gate
                         W2 core (8K)
                              │ gate
                         W3 frontier (16K)
                              │ gate
                  T1 single-turn tools ──► T2 multi-turn tools / search (SkyRL)
                              │ gate
                         F final selection, release evals

RL-0 — Think probe (done: job 61868620, 48 updates, 2 h 51 min, ~50 local-h)

Result (quick reasoning eval, held out, vs Stage-3): thinking accuracy 0.435 → 0.512, unfinished 48.9 % → 39.3 %, mean thinking length 3,478 → 2,970 tokens; default mode 0.581 → 0.593 (retained). Largest gains on EU-language math (0.108 → 0.308; unfinished 86 % → 58 %) and MGSM (0.59 → 0.66); MATH-500 flat (0.16 → 0.14, 80 % unfinished at 6K). Decision: closure responds to RL (+9.6 points closed, accuracy up) → proceed; run A2 because hard problems still do not finish; the update count is not capped by the episode counter (48 of 48 ran).

Follow-ups submitted 2026-10-23 (2 nodes each, ~50 local-h each, quick eval of step 48 queued automatically):

Results of RL-0b and A2a (quick reasoning eval, thinking mode, held out; 570 items):

Model Accuracy Unfinished Tokens GSM8K MGSM MATH-500 EU math Trace in prompt language Default-mode acc.
Stage-3 v1.1 0.435 48.9 % 3,478 0.673 0.590 0.160 0.108 24.8 % 0.581
RL-0 (48 updates) 0.512 39.3 % 2,970 0.727 0.660 0.140 0.308 1.1 % 0.593
RL-0b (RL-0 + 48) 0.532 37.4 % 2,848 0.700 0.680 0.180 0.367 0.4 % 0.593
A2a (−0.5 if unfinished) 0.523 34.0 % 2,649 0.687 0.690 0.210 0.300 1.5 % 0.575

Readings: more updates on the same small pool still add a little (+2 points) but are flattening; the non-stop penalty gives the shortest, most often finished traces and the best MATH-500 at equal update count to RL-0 (+1.1 points overall vs RL-0, +5 on MATH-500 with 120/100-item subsets, so within noise individually). Every RL variant reasons in English. Decision for W1: keep the non-stop penalty (−0.5); the language question is A1’s.

Language finding (lingua on the quick-eval thinking traces, 270 non-English rows): the share of traces in the prompt language fell from 24.8 % (Stage 3) to 1.1 % (RL-0). Stage 3 reasoned natively on the fully native oellm-math-eu prompts (cs, fi, hu, nl, sv 10/10, el 9/10) but those traces rarely closed; RL-0 answers the same prompts with English traces that close and score. MGSM prompts carry an English instruction, and Stage 3 already reasoned in English there. So part of RL-0’s EU-math gain is a switch to English: every later run must log the trace-language rate, and A1 decides whether the language term goes into W1–W3.

A1 result (quick eval, thinking mode, held out; lingua audit):

  Stage 3 RL-0 A1 (language reward on EU rows)
Overall accuracy 0.435 0.512 0.514
EU math (fully native prompts, 120): accuracy 0.108 0.308 0.192
EU math: unfinished 85.8 % 57.5 % 73.3 %
EU math: trace in prompt language 56 % (67/120) 2.5 % (3/120) 65 % (78/120)
MGSM (English instruction line) 0.590 0.660 0.720
MATH-500 0.160 0.140 0.230
GSM8K 0.673 0.727 0.687

Reading: the language reward works (native traces 2.5 % → 65 % on native prompts, above Stage 3) and overall accuracy is unchanged, but on the prompts where it binds, accuracy is 11.6 points below RL-0 (still +8.4 over Stage 3) because native traces finish less often (73 % unfinished). The cost is a termination weakness of native-language reasoning, which comes from mostly-English reasoning SFT data, not from the reward. Against the decision rule (cost ≤ 2 points ⇒ adopt), the language reward is not adopted for W1 as is. Options for the user/project to decide (roadmap §6): (a) English traces allowed, answers in the user’s language, legibility monitored; (b) add native-trace SFT (a “bridge” of verified native-language reasoning traces) before RL, then re-test A1; (c) language reward with a softer weight (e.g. correct but off-language = 0.5) to trade accuracy against native traces. Differences per benchmark are within noise at 100–200 items; the EU-math gap (37 vs 23 correct of 120) is the clearest effect.

Per-language breakdown of A1 (native EU math, ~8 prompts per language, so noisy): de/fr/es/en keep 0.68–0.80 accuracy with 16–26 % unfinished; cs/hu/nl/sv/bg/fi fall to 0.0–0.1 with 60–100 % unfinished at the 6,144-token limit. The cost is concentrated in the smaller languages and looks like a token-budget/termination problem rather than wrong reasoning. Check: the 114 unfinished native-language items re-generated at 16,384 tokens (job 62021559, eval/reasoning-quick/think-lang-step48-unfinished-native-16k). If most of them finish and are correct, native reasoning becomes a per-language budget/SFT question (option b) rather than a capability limit. 16K result (job 62244120, 10-25): with 16,384 tokens only 22 % of the 114 items finish and 8.8 % are correct (de/fr best: 33–44 % unfinished, 22 % correct; sv/nl 100 % unfinished at 16,384 tokens). Native traces run away or loop rather than running out of budget, so a larger budget does not rescue them; this supports option (a). Native reasoning needs good native-trace SFT data first.

Decision (2026-10-23): option (a) with an answer-language check. W1–W3 allow traces in any language; the EU-language math rows use multilingual_math with label "check": "answer" (oellm-rlvr f6552b6): reward 1 only if the answer is correct, the think/box form is valid and the final answer’s prose (after </think>, boxes and LaTeX removed) is in the prompt language; answers with < 20 letters of prose (a bare \boxed{}) pass. Pool: CHECK=answer scripts/leonardo_a1_language_data.sh → data/answer-language/ (2,008 rows, sha 0b745cd2…). On the existing quick-eval outputs, 92–96 % of finished non-English answers already pass (RL-0: 156/172, A2a: 183/192), so the term rarely binds and needs no separate ablation; it mainly stops two drifts seen in RL-0/RL-0b/A2a: English answers to German/French/Hungarian prompts and Russian answers to Bulgarian prompts. Native-language reasoning becomes a later opt-in (system-prompt switch, high-resource languages first) on top of the English-reasoning model.

RL-0 card (as planned)

   
Parent Stage-3 v1.1, RL thinking-default template
Question Does DAPO with “unclosed trace ⇒ reward 0” make traces close, without hurting accuracy?
Data 2,008 prompts: 1,408 oellm-math (multilingual) + 600 GSM8K
Settings DAPO, β 0, LR 1e-6 constant, 16 × 8, max response 6,144, total_episodes 6,144, 2 nodes
Cost ≤ 320 local-h (estimate ~140)
Live signals closed-trace rate per update, mean length, sequence_lengths_max hits, entropy, mixed-group fraction
End eval quick reasoning eval (thinking, 6K) vs Stage-3 v1.1; EUROPA Reasoning both modes
Decision closed rate up ≥ 10 points with accuracy flat or up → termination is an RL problem, proceed. Closed rate flat → add explicit length shaping (A2) before W1. Accuracy down → lower LR (5e-7) in all later runs.
Open item canary logged ~200 episodes per update (expected 128; dynamic-sampling refills are counted), so 6,144 episodes may give ~30 updates, not 48. Record the real update count; set later runs by max_updates, not episodes.

A-series — ablations (2 nodes each, on Stage-3 v1.1, in parallel with DPO)

Each runs 32–48 updates at 16 prompts × 8 samples, 6K max response, against the RL-0 settings as the baseline arm.

Status:

Run Status Result / note
A1 language (arm c, think-lang, job 61928425) ✅ done not adopted; answer-language check (arm b) adopted for W1–W3 without its own run (it binds on < 10 % of correct answers); 16K re-eval of unfinished native items queued (62021559)
A2a non-stop penalty (think-nonstop, job 61923004) ✅ done fewest unfinished (34 %), best MATH-500; adopted for W1–W3
A2b loop penalty not needed yet revisit if loops persist at 8K/16K
A3 both modes (think-mixed, canary 61969304 → full 61969305) ✅ done 10-24: training reward mcq_letter_direct 0.06 → 0.21, math_direct 0 throughout (the model never samples a bare answer; format DPO first). Quick eval: thinking 0.532 / unfinished 34 % (= best of A2a/RL-0b), default 0.593 (RL-0 0.593); mixing default-mode rows costs nothing; adopt per-row default-mode rows for W1 after the DPO format pass template templates/oellm_qwen3_two_stage_v2_1_rl_mixed.jinja (thinking by default, no thinking block for rows with rl_mode: default); verifiers math_direct / mcq_letter_direct (bare correct answer only, no think tags; OELLM_PATCH_DIRECT_MODE_VERIFIERS); pool oellm-rlvr/scripts/leonardo_a3_mixed_mode_data.sh = RL-0 rows + 430 math + 430 letter-only default-mode rows (letter rows mostly bg/hr: the format pairs used most other exams_qa rows); A2a recipe
A4 tool single-turn (think-tool, canary 61968632 → full 61968633) ✅ done 10-24: no effect — training reward 0.58 → 0.52; held-out tool decision 0.867 (Stage 3: 0.870), JSON 0.994; thinking 0.432 / unfinished 46 %. Not adopted; tool RL needs a harder/curriculum pool (T1) verifier tool_call (OELLM_PATCH_TOOL_CALL_VERIFIER): 0.2 well-formed + 0.3 tool names + 0.5 arguments, no-call rows 1 only without a call, scored after exactly one closed trace; pool scripts/leonardo_a4_tool_data.sh = 1,000 English grpo + 500 non-English single-call + 500 no-call rows from the Stage-3 tool dataset (held-out bucket excluded), tool blocks re-rendered with our template
A5 code (think-code, job 61949439) ✅ done 10-24 (walltime at ~step 30/48; evaluated step 24) training: all-tests pass 0.16 → ~0.30, truncated 73 % → ~30 %. EvalPlus HumanEval+/MBPP+ thinking 0.256/0.265 (Stage 3 0.189/0.243, RL-0 0.250/0.251), closed 57/61 %; direct 0.323/0.344 (unchanged). Math thinking 0.481. Small gain, mostly closure; code in W2 with a longer walltime (~8 h for 48 updates) plus a code-reasoning SFT top-up
Run Question Arms Data Cost Decides
A1 language Can RL make the model reason in the prompt language, and what does it cost in accuracy? (a) baseline; (b) answer-language multiplier; (c) answer + trace-language multiplier oellm-math EU24 + exam-MCQ, non-English only 2 × ~150 local-h (arm a = RL-0) whether W1–W3 use the trace-language multiplier, and for which languages (drop any language that loses > 3 points)
A2 length (only if RL-0 does not fix closure) Which shaping fixes termination? (a) DAPO soft overlong punishment; (b) (a) + loop penalty (n-gram repeat ratio ⇒ 0) RL-0 data 2 × ~150 local-h the structure guards for all windows
A3 both modes Does mixing default-mode prompts keep and improve the direct mode? (a) thinking only; (b) 70 % thinking + 30 % default-mode strict-format and numeric prompts math + strict-format set ~150 local-h (arm a = RL-0) the default-mode share in W1–W3; expected to fix EUROPA numeric
A4 tools single-turn Does the ToolRL reward on the 46K GRPO split improve held-out tool decisions? one arm oellm-eu-tooluse-v1 grpo + no-call items ~150 local-h whether tools enter W2 or wait for T1
A5 code Is there learning signal in single-turn code RL, and does it transfer to HumanEval+/MBPP+? one arm 2,004 oellm-code-rlvr train problems (12 families, d1–4) ~80 local-h whether code enters W2 as planned or needs an SFT top-up first

Ablation total: ~600–900 local-h.

W1 — Warm-up window (Stage-4 DPO checkpoint)

   
Question Does the RL-0 + ablation recipe transfer to the DPO checkpoint, at the production batch size?
Data math (oellm EU24 ≥ 40 % non-English, DeepMath 3–5), exam-MCQ (letter-only answers; only sources that are neither evaluation benchmarks nor license-unclear: drop belebele, global_mmlu, mmmlu, xcopa — we report them in oellm-eval — and every unknown/NC/needs-review source; today that leaves mainly exams_qa and slovak_financial_exam), strict format, IF ≤ 2 constraints; 30 % easy / 60 % medium / 10 % hard by measured pass@8
Settings 64 prompts × 8, max response 4K, 48 updates, decisions from A1–A3
Cost profiling ~150 + training ~450 local-h
Gate target holdouts up (paired test), closed rate ≥ 90 %, no retention regression

W2 — Core window

   
Data + DeepMath 5–7, POLARIS, PrismMath, oellm-code d1–3, IF ≤ 5 constraints, tool calls (if A4 passed); 10 / 55 / 35 %
Settings 64 × 8, max response 8K, 64 updates, re-profiled on the W1 export, fresh optimizer
Cost profiling ~250 + training ~1.5K local-h
Gate as W1, plus code holdouts and IFEval

W3 — Frontier window

   
Data + DeepMath 7–9, DeepCoder, code d4–5, hard multilingual math, re-profiled former 0/8 items; 5 / 45 / 50 %
Settings 64 × 16, max response 16K, 64 updates
Cost profiling ~400 + training ~4.5K local-h
Gate AIME / LiveCodeBench / PolyMath gains; closure at 16K ≥ 95 %

Scaling rule: if a window’s holdout curve is still rising at its last checkpoint, extend it in steps of 32 updates from the reserve rather than starting the next window.

T1 — Single-turn tool RL (if not already in W2)

oellm-eu-tooluse-v1 grpo (46K, English only, exactly one call per row, no no-call rows — checked 10-23) plus multilingual and no-call rows from the Stage-3 tool dataset (as in A4), Nemotron function-calling pivot; ToolRL reward; default and thinking mode. ~800 local-h. Gate: held-out tool eval decision ≥ 0.92, BFCL irrelevance, no regression.

T2 — Multi-turn tools and search (SkyRL)

Mock-tool loops with injected tool errors (the EUROPA tool-recovery gap), conversational tool use with simulated users, and a search agent over a local multilingual Wikipedia index (the European angle: retrieval in the user’s language). Needs SkyRL qualified on Leonardo first (it is qualified on LUMI); qualification canary ~100 local-h. Training ~3–5K local-h. Gate: EUROPA tool-recovery ≥ 0.8, τ²-bench (eval only), no retention regression.

F — Final selection and release

Full eval suite on the last two promoted checkpoints (both modes), the special-token check, the identity audit, and a fresh-seed EUROPA Reasoning variant (--seed) as the uncontaminated release check. ~300 local-h.

5. Budget summary

Block Local-h (estimate)
RL-0 think probe ~140–320
A-series ablations ~600–900
W1 / W2 / W3 (incl. profiling) ~600 / ~1.75K / ~4.9K
T1 / T2 (incl. SkyRL canary) ~800 / ~3–5K
Evaluation across the program ~1.5K
Final selection ~300
Total ~14–16K, leaving ~9–11K of the 25K envelope for window extensions

6. Live monitoring and stop rules (all runs)

Stop the run (and keep the last good checkpoint) if any of these hold for 4 consecutive updates:

Signal Stop when
Entropy falls below 30 % of its first-update value (collapse)
Mean response length rises > 50 % with no reward gain (padding / looping)
Hits at max length > 25 % of rollouts
Mixed groups < 20 % of groups have mixed rewards (no learning signal; re-profile)
Default-mode think tags any
Verifier errors > 2 % of rollouts
Language (A1 onward) trace-language rate falls > 10 points

Every window also gets a manual audit of the 20 highest-reward rollouts (reward hacking: answers inside the trace, several boxed answers, prompt echo).

7. Merging and combining checkpoints

Merging is cheap (minutes on CPU with mergekit, no GPU) and works when the models share a parent, tokenizer and template and differ by small deltas, which holds for our DPO and RL stages (LR 5e-7 to 1e-6). The cost is the evaluation, so a merge is only made where it answers a concrete question.

Standing (every run):

When Merge Compared against Cost
End of DPO and of every RL window uniform average of the last 2–3 saved checkpoints the best single checkpoint one eval pass
After DPO, only if a gate shows thinking-mode or tool regression linear interpolation Stage-3 v1.1 → DPO, α ∈ {0.5, 0.75, 1.0} (α = 1 is DPO) DPO and Stage 3 three eval passes

The DPO interpolation is the first fallback for the risk that all-default-mode, tool-free DPO pairs weaken thinking or tool use; it replaces “start RL from Stage 3” unless every α fails the gates.

Options (used only if the situation arises):

Never merge: models from a different base, tokenizer or RoPE configuration (use distillation instead); models trained for different template contracts (e.g. Stage 2 with Stage 3); Stage-3 v1 (0dc97efa), whose collapsed tag rows would partly undo the v1.1 repair.

Rules for every merge: the output keeps our tokenizer and the v2.1 template (never mergekit’s tokenizer union); mixing weights are chosen on the selection holdouts, never on the release evaluations; the merged checkpoint passes the same gates as any checkpoint (both modes, tool eval, EUROPA Eval/Alignment/Reasoning, trace closure, tag-row check) and is recorded with its recipe (inputs with revisions, method, weights, mergekit version).

8. Order of work

Done: RL-0 → RL-0b, A2a, A1 (+16K re-eval), A3, A4, A5 (results in §4 and the status table).

Queued 10-25 evening (from the Stage-4 candidate = linear merge 0.5 × Stage-3 v1.1 + 0.5 × DPO v2, model dir models/stage4-merge-a05-mixed = merge weights + RL mixed template). The P1/A4b runs planned from DPO v2 were cancelled before they started once the merge restored tool use (roadmap §5).

Next:

  1. ✅ Language policy decided (A1 section): traces any language, answer in the prompt language. ✅ 16K re-eval done.
  2. A5 result + EvalPlus on its final export → decide code in W2 (or SFT top-up).
  3. Build A3 (per-row default mode) and A4 (tool reward); run both on Stage-3 v1.1 while DPO trains.
  4. Build W1 pools: oellm-math-rlvr EU24 (≥ 40 % non-English) + DeepMath 3–5, exam-MCQ (35 languages, letter answers), strict-format set, IF ≤ 2 constraints; decontaminate against every eval; profiling job (pass@8) ready to run on the DPO checkpoint the day it passes its gates.
  5. W1 → W2 → W3, each gated; T1 inside W2 if A4 passes; legibility and language monitors from W1 on.
  6. In parallel with W2/W3: SkyRL qualification canary on Leonardo, mock-tool and Wikipedia-search environments.

9. Run card template

Run ID / job IDs:
Parent checkpoint (path, HF revision):
Question (one sentence):
Data: pools, rows, languages, realized mix (logged), decontamination report
Reward: verifier versions, multipliers, guards
Settings: algorithm, LR, prompts x samples, max response, updates, nodes
Budget: estimate (formula §2) and actual
Live stop rules: §6 (+ run-specific)
End evaluation: holdouts and retention suite, both modes
Decision rule: what result leads to which next run
Result: numbers, selected checkpoint, decision taken

10. Continual improvement loop (from W4)

Status 2026-10-29. After W3 the model improves in repeated rounds instead of a fixed run sequence. More data is not the constraint (open verifiable data and our own generators are effectively unlimited); difficulty fit, coverage and release gates are. W2 left 69 % of its prompts unsolved at k = 8 (no GRPO signal), so every round re-profiles its pool on the checkpoint it will train.

10.1 One round

  1. Measure the current checkpoint: the eval suite (knowledge, EU task suites, IF, tools, code, identity, safety, both modes) plus environment baselines: svenska-rl (slurm/svenska_rl_baseline_leonardo.sbatch) and NeMo Gym (slurm/nemo_gym_baseline_leonardo.sbatch).
  2. Diagnose 2–3 weak areas from per-task scores, solve rates and the previous run’s filtered/unsolved ratios.
  3. Build the pool by pass rate on that checkpoint (oellm-rlvr pass-rate-sample/-score/-select, scripts/leonardo_pass_rate.sbatch): k = 8 samples with the RL template, the training verifiers per dataset (same OELLM_PATCH_* flags), keep prompts whose reward varies within the group (optionally a solve-rate band), cap per dataset, decontaminate against every eval. Record the profile and the model in the pool manifest.
  4. Train one RL window (~150–250 updates, the W2 recipe: DAPO, async, non-stop and length penalties). Behaviours RL cannot reach go to targeted DPO/SFT (identity, safety: Stage 4b).
  5. Gate the release: the targeted areas improve and nothing on the guard set (knowledge, identity, safety, IF, multilingual, both modes) regresses beyond its margin. Then model card, HF/Ollama release.
  6. Record the round in this document (run card §9) and update the next round’s candidates.

10.2 Candidate data

Source Adds Status / requirement
EU task suites (eu_qa, eu_doc) calibrated knowledge with UNKNOWN, EU document tasks in W3
svenska-rl (8 environments, contract v0.2) Swedish office work: invoices, payment matching, scheduling, redaction, procurement, stock baseline done (below); typed ones need a svrl_typed verifier in compat
Translated verifiable prompts (math, logic, tool tasks into the 12 EU languages) multilingual reasoning; answers language-independent, verifiers unchanged qwen3.6 translation pipeline exists; spot-check
NVIDIA Nemotron-RL (CC-BY/ODC-BY): instruction following, Structured Outputs, QA-Abstention, SysBench, ReasoningGym schema/format compliance, system-prompt following, abstention against hallucination NeMo Gym baseline in progress; judge-based ones use the local Gemma judge
Procedural puzzles (ReasoningGym-style) logic, planning, counting with difficulty knobs unlimited
Long-context synthetic tasks (32K–128K retrieval/aggregation) uses the 256K context in RL needs long-rollout throughput tests
Multi-turn tool environments (svenska-rl tool episodes, Workplace Assistant) agentic work over several steps needs a multi-turn RL runtime (SkyRL/Harbor adapter or NeMo RL); T1 showed single-call tool RL saturated
Rubric-judged writing (EU-language e-mails, summaries, plain language) quality without an exact answer later; reward-hacking audit first

10.3 Order

Round Content Requirement
W3 EU task suites + the W2 mix queued (2 nodes); move to 1 node if it keeps waiting
Stage 4b DPO identity (v2.1) + safety pairs (reduced: ~2,000 prompts × 4 samples), both sampled on the W3 checkpoint (on-policy; the 711 W2 identity pairs are a pilot) after W3 lands (decided 10-29); gate on EUROPA refusal calibration (0.35) and grounding & safety (0.60), which regressed in the merge, not in RL
W4 + Structured Outputs, QA-Abstention, SysBench, svenska-rl typed tasks, translated math/logic; pool by pass rate svrl_typed verifier patch; baselines
W5 multi-turn tool RL (svenska-rl tool episodes, Workplace Assistant) multi-turn runtime
W6 long-context RL throughput tests
later rubric RL for writing quality judge audit

Cost: one 2-node window of ~24 h ≈ 190 GPU-h ≈ 1.5K local-h; measurement and pool building add ~100–200 local-h per round. Compute is not the bottleneck (§2); curation and the multi-turn runtime are.

10.4 First measurements

svenska-rl L1 baseline, W2 (= Agora 9B), 64 eval tasks per environment, greedy, job 62626669 (17 min, 1 GPU), svenska-rl a8a81dc; re-scored after the answer-parsing fix in the next commit (JSON keys without diacritics, markdown around SVAR:).

Environment Mean reward Solved Note
document_redaction 0.79 0.25 finds most items; exact lists are rare
payment_matching 0.66 0.36 good candidate band
inventory_reorder 0.21 0.00 answers 0 for every article: equals the all-zero shortcut (0.22)
payment_registration (tools) 0.15 0.02 often stops after one tool call
invoice_amount_due 0.11 0.11 arithmetic over many lines, rounding to whole kronor wrong
meeting_slot 0.11 0.11  
supplier_quotes 0.09 0.09  
meeting_reschedule (tools) 0.06 0.03  

Most environments sit in or near the useful band for GRPO; the low ones need a pass-rate profile at k = 8 (greedy accuracy understates sampled variance) before they go into a pool.

10.5 Reward-coverage bug in W1a/W2 (found 10-29, fixed before W3)

Building the pass-rate filter showed that several verifier patches never took effect in training. Each OELLM_PATCH_* flag installed its own post-import hook on open_instruct.ground_truth_utils; the first hook on sys.meta_path loaded the module and ran only its own patch, so only the last-registered verifier patch applied. Separately, ifeval_think subclassed ifeval_direct and build_all_verifiers only instantiates direct subclasses. Training logs (“No reward function found for dataset …”): W1a skipped multilingual_math (6,305); W2 skipped ifeval_direct, ifeval_think, math_direct_shaped, mcq_letter_direct, multilingual_math (22,993), and the closed-trace check was off. Those rows got no task reward, so their groups mostly carried no signal: wasted samples, not wrong gradients (the non-stop penalty still applied). W2’s “69 % unsolved” includes these unscored rows and overstates the difficulty problem; re-measure with the pass-rate filter. Fixed in oellm-rlvr bfcdb83 and 089ce7b (regression tests); a compute-node check and the pass-rate scoring smoke (job 62637819: all W3 datasets scored, 0 errors) passed, then W3 (62430101) was released. W3 is the first window with every verifier active.