# Claim 4: ablations (ICL seeding, novelty sharpening)


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_5d2087e2594f", "created_at": "2026-07-16T18:28:57+00:00", "title": "Method summary for this claim"}
-->
## Method summary for this claim

Direct analog of the paper's Figure 3b: compares `xrpo_full` against `xrpo_no_icl` (ICL seeding disabled) and `xrpo_no_novelty` (novelty sharpening disabled), all else equal (same steps, data, seed budget). Since our full XRPO only reimplements 2 of the paper's 3 mechanisms (no adaptive rollout allocator), this ablation isolates exactly those 2 components against each other and against the no-mechanism GRPO baseline.


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_4a21a99f776a", "created_at": "2026-07-17T04:35:59+00:00", "title": "Result: Claim 4 partially testable, and NOT reproduced where it was"}
-->
## Result: Claim 4 partially testable, and NOT reproduced where it was

**Novelty-Guided Advantage Sharpening ablation: not testable.** `xrpo_full` and `xrpo_no_novelty` produced bit-identical training trajectories and final metrics (see the Overview & Scope bug note) — the novelty-sharpening code silently never executed because TRL's `old_per_token_logps` is `None` under on-policy (`num_iterations=1`) GRPO in TRL 1.8.0, and our implementation only applies the bonus when that tensor is present. This mechanism was not actually exercised in this reproduction; no conclusion can be drawn about the paper's claimed 3.5% pass@1 decline from removing it.

**ICL Seeding ablation: testable, and shows the opposite direction from the paper.**

| config | avg pass@1 | avg cons@8 |
|---|---|---|
| xrpo_full (ICL on) | 0.040 | 0.010 |
| xrpo_no_icl (ICL off) | 0.020 | 0.000 |

Removing ICL seeding *decreased* pass@1 (0.040 → 0.020) and cons@8 (0.010 → 0.000) here — consistent in *direction* with the paper's claim that removing ICL Seeding hurts performance (paper: 4.2% cons@32 decline). This is the one result in this reproduction that points the same way as the paper, though the absolute magnitude (2pp on a 50-question single/8-sample eval) is well within noise for a single run and should not be over-read. `xrpo_full`'s ICL bank grew to 168 solved exemplars with 2432 total injections across 80 steps (heavy reliance on ICL seeding at this scale, since GSM8K questions are frequently missed by a barely-trained 0.6B model — most groups start at zero reward, triggering the zero-reward-group condition often).