# Claim 1: pass@1 vs GSPO
---
## Method summary for this claim
**pass@1 proxy**: on our scaled setup, one sampled rollout per eval question (of the `num_generations=8` sampled), averaged over GSM8K-test (60) + MATH-500 (60) subsets, for the final XRPO-full vs GRPO-baseline checkpoints after 100 GRPO optimizer steps on Qwen3-0.6B.
Caveats vs. the paper's Table 1: single benchmark family (no AIME/HMMT/BRUMO/Codeforces/LiveCodeBench), much smaller model (0.6B vs 1.7B), far fewer optimizer steps, MATH-500 grading uses simple boxed-answer string/numeric matching (slightly noisier for non-numeric answers like coordinate tuples) rather than the paper's likely symbolic equivalence checker. Treat this as a directional relative comparison (does XRPO's mechanism combination help over GRPO at this scale), not an absolute match to the paper's numbers.
---
````html
````
````raw
config,gsm8k_pass1,gsm8k_cons8,gsm8k_avg_len,math500_pass1,math500_cons8,math500_avg_len,avg_pass1,avg_cons8,avg_len,icl_bank_size,icl_injections_total,elapsed_min
grpo,0.1,0.02,361.0425,0.0,0.0,372.5825,0.05,0.01,366.8125,,,83.83463437557221
xrpo_full,0.08,0.02,382.5175,0.0,0.0,383.995,0.04,0.01,383.25625,168.0,2432.0,173.08483233849208
xrpo_no_icl,0.04,0.0,382.6475,0.0,0.0,383.9825,0.02,0.0,383.315,107.0,0.0,83.17931364774704
xrpo_no_novelty,0.08,0.02,382.5175,0.0,0.0,383.995,0.04,0.01,383.25625,168.0,2432.0,173.05054400364557
````
---
## Result: Claim 1 NOT reproduced at this scale
Final avg pass@1 across GSM8K-test (50) + MATH-500 (50) subsets, single sample per question:
| config | gsm8k pass@1 | math500 pass@1 | avg pass@1 |
|---|---|---|---|
| grpo (baseline) | 0.10 | 0.00 | **0.050** |
| xrpo_full | 0.08 | 0.00 | 0.040 |
| xrpo_no_icl | 0.04 | 0.00 | 0.020 |
| xrpo_no_novelty | 0.08 | 0.00 | 0.040 |
XRPO's implemented mechanisms (ICL Seeding; novelty sharpening did not actually run — see Overview & Scope bug note) did **not** beat the GRPO baseline on avg pass@1 at this scale — the opposite direction from the paper's claimed +4pp pass@1 over GSPO (note: we compare to GRPO, not GSPO — GSPO was not reimplemented, a further scope reduction).
MATH-500 pass@1 is 0.00 across every config — the 0.6B model with ≤80 GRPO steps essentially never solves a MATH-500 problem in one sample; this benchmark is not discriminating at this scale and should be discounted (also affected by the imperfect exact-match grader for non-numeric answers, see Overview & Scope).
**Bottom line**: this scaled reproduction does not show XRPO's pass@1 advantage. Plausible reasons, not disentangled here: (1) novelty sharpening never activated (a real component missing); (2) TF-IDF-based ICL retrieval is a much weaker substitute for Qwen3-Embedding-8B and may inject irrelevant/noisy exemplars; (3) 0.6B/80-step/8-rollout training is far too small a regime for either mechanism's benefits (which the paper demonstrates over 1.7B/~1000+ steps/128 rollouts) to manifest; (4) eval sample sizes (50/benchmark, 1 sample/question) are too small to resolve small effect sizes.