FastWAM GRPO RL β LIBERO-10 (round15 start)
Critic-free flow-SDE GRPO fine-tune of a FastWAM
policy (β1B action expert trained; 5B Wan2.2 video expert frozen) on the LIBERO-10
(Long) suite, starting from the round15 SFT checkpoint. Trained with
RLinf (FSDP2, 4ΓA100), grpo_norm_by_std=false
(Dr.GRPO), noise_level=0.6, lr 2e-5, clip 1.0, 256 traj/step, 10 steps.
dcp_checkpoint/ is a Torch Distributed Checkpoint (model + optimizer), resumable via
RLinf runner.resume_dir.
LIBERO-130 eval (RL step10 vs base round15)
Deterministic full eval β all 50 init-states/task, per-suite max-horizon, 100 parallel envs, zero measurement variance. RL was trained only on libero_10; the other suites test generalization / forgetting.
| Suite | tasks | base SR | RL SR | Ξ |
|---|---|---|---|---|
| Spatial | 10 | 0.962 | 0.972 | +0.010 |
| Object | 10 | 0.972 | 0.912 | β0.060 |
| Goal | 10 | 0.956 | 0.962 | +0.006 |
| Long (libero_10, trained) | 10 | 0.852 | 0.906 | +0.054 |
| Short (libero_90) | 90 | 0.914 | 0.850 | β0.065 |
| LIBERO-130 (task-weighted) | 130 | 0.921 | 0.877 | β0.044 |
GRPO on libero_10 gives a real, headroom-driven gain on the trained suite (+0.054) and holds spatial/goal, but the action expert specializes toward libero_10's long-horizon behavior and mildly forgets object (β0.060) and the saturated libero_90 (β0.065; 54/90 tasks worse). Since libero_90 is 90 of the 130 tasks, forgetting dominates the aggregate.
Full per-suite/per-task numbers and reproduction scripts: wzii/RLinf Β· experiments/fastwam_rl_libero10.