sandheepp/qwen3.5-0.8b-grpo-musique-h200-m5_5-seed42-f1-floor-fmt

Qwen3.5-0.8B GRPO on MuSiQue (F1 + 0.1 floor + format-gate reward) — H200 M5.5 seed42

GRPO-trained checkpoints of Qwen3.5-0.8B on MuSiQue (multi-hop QA) with a Wikipedia FAISS retriever, F1 + 0.1 partial-credit floor + format-gate reward, and the ReSearch-paper recipe (arXiv:2503.19470) ported to NeMo-RL.

This run is the reward-shape ablation of pantomiman/qwen3.5-0.8b-grpo-musique-h200-a4-seed42-f1-only — same hardware (H200 Spheron), same hyperparameters, only the reward function changes.

⏹ Status: ENDED at step_209 (67 % of MuSiQue epoch 1)

Last checkpoint: step_200/ (last cadence-aligned save; training reached step 209 before the H200 instance was killed when credits ran out 2026-05-18). Run is permanently ended at step 209/311; not resumable in-place because the persistent volume was deleted at credit-exhaustion. All training data (20 checkpoints, 209 rollout jsonls, full prod.log) is preserved here on HF — that survived even when the volume was wiped.

W&B run: gtf8xe1d (project name has leftover _b300_ prefix from the b300-derived config; functional but not renamed mid-run).

What's in this repo

Path	Contents
step_10/, step_20/, ..., step_200/	One checkpoint per 10 GRPO steps. Each contains policy/weights/model/consolidated/ (safetensors), policy/tokenizer/ (Qwen3.5 tokenizer), config.yaml (training config snapshot at that step), training_info.json, train_dataloader.pt. ~6.4 GB each. step_200/ is the final checkpoint. Total: 20 checkpoints, ~128 GB.
config_snapshot.yaml	The launch-time training config (also reproduced inside each step_N/).
logs/prod.log	Full training log (NeMo-RL output, step timing, reward summaries).
rollouts/train_data_step1.jsonl through rollouts/train_data_step209.jsonl	Per-step rollout corpus, 209 files, no gaps. 320 rollouts per step (= 40 prompts × 8 GRPO group samples), each with the full message log (<think>, <tool_call>, <tool_response>, <answer>) + per-rollout reward. JSONL, ~80-120 MB / step.

Everything needed to reproduce, audit, or fine-tune from any checkpoint is included. No GPU required to read or analyse this repo; only checkpoint inference + training need a GPU.

Quickstart — load a checkpoint for inference

from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "sandheepp/qwen3.5-0.8b-grpo-musique-h200-m5_5-seed42-f1-floor-fmt"
STEP = "step_200"  # any of step_10 ... step_200

model = AutoModelForCausalLM.from_pretrained(
    f"{REPO}/{STEP}/policy/weights/model/consolidated",
    torch_dtype="bfloat16",
    device_map="auto",
)
# Tokenizer is identical across checkpoints; use the base Qwen3.5-0.8B tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")

The model was trained to emit <tool_call> blocks for retrieval and <answer>...</answer> blocks for the final answer. Match that scaffold at inference. The Qwen3.5 native tool-call template is the post-training format the model was aligned to; use that.

Training recipe (M5.5)

Knob	Value
Base model	Qwen/Qwen3.5-0.8B (hybrid; GatedDeltaNet + attention; 248K vocab; sm_90 / Hopper)
Training corpus	MuSiQue (19,938 multi-hop QA training rows; single-dataset; hardest of the four ReSearch-paper benchmarks)
Retriever	Wikipedia-2018 corpus, E5-base-v2 embeddings, FAISS IVF4096-SQ8 quantised index, 4-worker CPU service on port 3005, retrieval_topk=5
Reward	F1 + 0.1 floor + format-gate. Returns 0 if no <answer>...</answer> tag (format-gate); else max(0.1, token_F1(answer, gold)) (0.1 floor for format-valid-but-F1=0). The ablation knob vs M5.1's pure F1.
Algorithm	GRPO, KL coefficient = 0.001, group-relative advantage with no normalisation, ratio_clip=0.2
Batch shape	num_prompts_per_step=40 × num_generations_per_prompt=8 = 320 rollouts/step
Sequence budget	max_total_sequence_length=8192 tokens
Rollout	up to 10 search-tool turns per question, observation truncated to 256 tokens / chunk
Optimizer	AdamW, lr=1e-6, weight_decay=0.01, temperature=1.0
Memory knobs	train_micro_batch_size=2, activation_checkpointing=true, vLLM gpu_memory_utilization=0.5, async engine, sleep mode
Schedule	311 steps = 1 epoch (config max_num_steps=311); reached 209 before credit-exhaustion shutdown
Checkpoint cadence	every 10 steps (M5.5 tightened from M5.1's 50)
Eval cadence	none in-loop; checkpoints are evaluable offline via evaluation_qwen35

Hardware + cost

GPU	1× NVIDIA H200 SXM5 (141 GB VRAM, sm_90 / Hopper)
Provider	Spheron Network
Tier	Dedicated throughout
Step wall (cadence-dependent, 2.3-18 min)	C8-C9 = 4 min (efficient regime); C13-C14 = 9-15 min (over-search drift); cold C1 = 18 min
Cost per step	~$0.15 / step (efficient) to $1.20 / step (over-search drift)
Total spent through step_209	~$80-$100 (rough; Dedicated throughout)
Persistent volume	virtiofs milestone55 (500 GB) — deleted at credit-exhaustion. HF Hub was the durable backup that survived.

Mandatory vLLM patch for Hopper (sm_90)

vLLM 0.17.1 + FlashInfer 0.6.4 routes Qwen3.5's GatedDeltaNet through a FlashInfer GDN kernel on Hopper that deadlocks on multi-turn prefill at production batch sizes (~60 min stall, GPU idle, no error). Patch vllm/model_executor/models/qwen3_next.py:156 to force the native Triton path:

# Apply to both sync and async vLLM venvs
sed -i 's|if current_platform.is_cuda() and current_platform.is_device_capability(90):|if False:  # PATCHED Hopper deadlock|' \
  path/to/site-packages/vllm/model_executor/models/qwen3_next.py

The native Triton path is functionally correct; only the FlashInfer fast-path is deadlock-prone on Hopper at this configuration. All checkpoints in this repo were produced with the patch applied.

Live trajectory

10-step cadences. Each row is a mean over 3,200 rollouts (10 steps × 320 rollouts/step).

Cadence	Steps	rew mean	rew > 0¹	tool mean	completion %	step wall	floor%	partial%	perfect%
C1	1-10	0.110	56 %	3.88	68 %	~600-1200 s (cold)	81 %	7 %	2 %
C2	11-20	0.160	99 %	1.15	100 %	~130-200 s	90 %	6 %	4 %
C3	21-30	0.189	100 %	1.10	100 %	~130-150 s	78 %	15 %	6 %
C4	31-40	0.221	99 %	2.55	100 %	~250-380 s	73 %	18 %	9 %
C5	41-50	0.233	99 %	2.60	100 %	~320-450 s	83 %	11 %	5 %
C6	51-60 (over-search peak 1)	0.258	99 %	4.30	99 %	~440-1085 s	63 %	13 %	18 %
C7	61-70	0.258	100 %	3.38	100 %	~270-440 s	73 %	14 %	12 %
C8 (efficient low 1)	71-80	0.269	100 %	2.14	100 %	~240-290 s	74 %	17 %	9 %
C9	81-90	0.290	100 %	2.21	100 %	~250-290 s	70 %	19 %	12 %
C10	91-100	0.314	100 %	2.53	100 %	~350-420 s	72 %	16 %	11 %
C11 (1st cycle high)	101-110	0.328	99 %	3.21	99 %	~520-700 s	62 %	19 %	18 %
C12 (drift begins)	111-120	0.305	100 %	3.30	100 %	~440-715 s	—	—	—
C13	121-130	0.277	100 %	3.65	100 %	~450-705 s	70 %	15 %	15 %
C14 (drift halt)	131-140	0.288	100 %	4.22	100 %	~715-895 s	76 %	15 %	9 %
C15 (recovery w/o compression)	141-150	0.297	100 %	4.33	100 %	~720-895 s	71 %	20 %	8 %
C16 (2nd compression starts)	151-160	0.332	99 %	3.89 ↓	100 %	~650-790 s	—	—	—
C17	161-170	0.334	99 %	4.01	99 %	—	—	—	—
C18 (cadence-mean run high)	171-180	0.343	99 %	4.33	99 %	—	—	—	—
C19 (single-step run high)	181-190	0.342	99 %	3.71 ↓	99 %	—	—	—	—
C20	191-200	0.336	99 %	3.81	99 %	—	—	—	—
C21 (partial, run ended)	201-209	0.325	99 %	3.75	99 %	—	—	—	—

¹ rew > 0 means the rollout was format-valid (got at least the 0.1 floor); for M5.5 this is effectively the format-mastery rate. The interesting split is floor% vs partial+perfect% (the latter = real F1 above the floor). The partial+perfect band grew from 9 % (C1) to 37 % (C11 peak) and held in the 24-39 % range through C12-C19.

Single-step run highs (top 15 of all 209 prod steps):

Rank	Step	Reward	Cadence
1	189	0.4217	C19
2	105	0.4190	C11
3	116	0.4112	C12
4	156	0.4059	C16
5	179	0.4011	C18
6	170	0.3989	C17
7	194	0.3881	C20
8	198	0.3703	C20
9	187	0.3691	C19
10	173	0.3689	C18
11	102	0.3672	C11
12	205	0.3637	C21
13	191	0.3633	C20
14	174	0.3627	C18
15	180	0.3606	C18

10 of the top 15 landed in C16-C20 (the 2nd-cycle break-out window). C18 alone contributed 4 entries.

What the policy learned (through C21)

By cadence 11 (step 110), the policy had acquired three durable capabilities; the C12-C21 cadences extended each:

1. Format mastery. Completion rate hit 100 % by step 20 and held throughout. The format-gate component of the reward is satisfied universally.
2. Iterative search. After an over-correction to single-shot answers in C2 (tool calls 1.15 — the model briefly tried to farm the 0.1 floor without searching), the model rediscovered multi-turn search by C4 (2.55) and stayed in iterative-search mode.
3. Reward growth via two complete drift-and-recover cycles.
   • Cycle 1 (C5 → C11): drift up to over-search peak C6 (4.30 tool calls), compress to C8 low (2.14), recover to cadence-mean high C11 (0.328).
   • Cycle 2 (C12 → C19): drift up to peak C14-C15 (4.22-4.33), recover via compression C19 (3.71), reach cadence-mean high C18 (0.343) and single-step all-time high step 189 (0.4217).

The two cycles have the same shape but the second peaked higher: cadence-mean 0.343 vs 0.328, single-step 0.4217 vs 0.4190. Tool-call median converged to ~3-4 across both cycles. Exploration excursions self-correct without intervention — the cost-adjusted reward optimum is a stable attractor.

Comparison vs M5.1-H200 (sibling, pure F1 reward)

Run is the A/B ablation against pantomiman/qwen3.5-0.8b-grpo-musique-h200-a4-seed42-f1-only. Same hardware, same hyperparameters, only the reward changes. Headline comparison (cadence-mean reward at matched steps):

Cadence	Steps	M5.1 (F1-only)	M5.5 (F1 + 0.1 floor + format-gate)	Δ
C4	31-40	0.171	0.221	+0.050
C8	71-80	0.221	0.269	+0.048
C11	101-110	0.280	0.328	+0.048
C14	131-140	0.240	0.288	+0.048
C16	151-160	0.256	0.332	+0.076
C18	171-180	0.275	0.343	+0.068

M5.5 cadence-mean is consistently +0.04 to +0.08 above M5.1 at matched steps. Some of that is floor-padding (a format-valid F1=0 rollout earns 0.1 in M5.5 vs 0.0 in M5.1) but the gap widens in the later cadences (C16-C18), where M5.5 also has higher partial+perfect% — so the gap is at least partly real F1 capability, not just floor mechanics.

Best per-cadence comparison: single-step run high 0.4217 (M5.5, step 189) vs 0.355 (M5.1, step 105 + step 170 tied). M5.5 single-step max is +19 % above M5.1's. The 0.1 floor + format-gate appears to accelerate the climb to high-quality F1 wins.

Reproducibility

Full setup runbook (Spheron H200 specifics + the FlashInfer GDN patch + the volume mount + the cadence cycle): docs/setup/B300_RUNBOOK.md. (The runbook was authored during M5.5's earlier b300 attempt and now covers both b300 and h200 paths.)

Cadence-cycle handoff (10-step analyzer + commit + push workflow): docs/milestone_5/CADENCE_HANDOFF_M5_5.md.

Training code (NeMo-RL @ v0.6.0 + the Qwen3.5 / search-tool overlay): training_m5_5/.

Reward function (40 lines, F1 + 0.1 floor + format-gate): training_m5_5/src/rewards/search_r1.py.

Training config used to launch this run is reproduced at the repo root as config_snapshot.yaml. The same file is embedded inside each step_N/ directory (config.yaml) for per-checkpoint reproducibility.

Internal cadence narrative (live trajectory with per-cadence commentary across all 21 cadences): docs/report/RESULTS_M5_5_H200.md.

Citation

If you use these checkpoints or the analysis above, please cite this run:

@misc{qwen35-08b-grpo-musique-h200-m5_5-f1-floor-fmt-2026,
  title  = {Qwen3.5-0.8B GRPO on MuSiQue (M5.5 H200 seed42, F1 + 0.1 floor + format-gate reward)},
  author = {Padmanabhan, Sandheep},
  year   = {2026},
  howpublished = {HuggingFace},
  url    = {https://huggingface.co/sandheepp/qwen3.5-0.8b-grpo-musique-h200-m5_5-seed42-f1-floor-fmt}
}

The accompanying paper is in preparation; this entry will be updated once it is available.