# Claim 2: cons@32 vs GSPO


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_a5428a2afbef", "created_at": "2026-07-16T18:28:56+00:00", "title": "Method summary for this claim"}
-->
## Method summary for this claim

**cons@8 proxy** (paper reports cons@32; we use 8 samples/question, matching our reduced `num_generations` rollout budget) computed as majority vote over the same 8 sampled completions per eval question, averaged over GSM8K-test (60) + MATH-500 (60).

Note: the paper compares against GSPO specifically for Table 1; we only implement/compare against a vanilla GRPO baseline (GSPO is a separate, more involved sequence-level importance-sampling variant not reimplemented here — out of scope for this reproduction). Results below should be read as XRPO vs GRPO, a weaker/adjacent comparison to the paper's XRPO vs GSPO.


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_c443109ee806", "created_at": "2026-07-17T04:35:35+00:00", "title": "Result: Claim 2 NOT reproduced at this scale"}
-->
## Result: Claim 2 NOT reproduced at this scale

Final avg cons@8 (majority vote over 8 samples/question — paper uses cons@32; we use 8 to match our reduced rollout budget) across GSM8K-test (50) + MATH-500 (50):

| config | gsm8k cons@8 | math500 cons@8 | avg cons@8 |
|---|---|---|---|
| grpo (baseline) | 0.02 | 0.00 | **0.010** |
| xrpo_full | 0.02 | 0.00 | 0.010 |
| xrpo_no_icl | 0.00 | 0.00 | 0.000 |
| xrpo_no_novelty | 0.02 | 0.00 | 0.010 |

XRPO ties GRPO on cons@8 (both 0.010) rather than beating it — no evidence of the paper's claimed up-to-6% cons@32 gain over GSPO. Also note we compare to GRPO, not GSPO (not reimplemented — scope reduction). At these near-floor accuracy levels (1-2%), cons@8 is barely distinguishable from cons@1 (pass@1) since almost no question has more than 1-2 of 8 samples correct; a genuine reproduction of this claim needs a training/model regime where baseline accuracy is well above the noise floor, which 80-step/0.6B training does not reach.