Round 8 gap-fill artifact: ROUND8_REPORT.md

ROUND8_REPORT.md

@@ -0,0 +1,54 @@
+# Round 8 — Fill R6 Gaps
+
+R8 fills three gaps R6 left open: K-sweep at N=24, per-tensor and Procrustes methods, and a per-task breakdown.
+
+## 1A. K-sweep at N=24
+
+| K | aggregate gap_recovered ± std |
+|---:|---:|
+| 2 | 0.032 ± 0.251 |
+| 4 | 0.069 ± 0.210 |
+| 6 | 0.083 ± 0.136 |
+| 8 | 0.129 ± 0.156 |
+| 12 | 0.131 ± 0.118 |
+| 16 | 0.129 ± 0.097 |
+| 20 | 0.130 ± 0.089 |
+| 24 | 0.126 ± 0.089 |
+
+No K beats R6 global_ridge at N=24 (0.135); K=12 is best among Top-K rows at 0.131.
+
+## 1B. Per-tensor methods
+
+| method | N=12 | N=16 | N=24 |
+|---|---:|---:|---:|
+| mean | 0.077 ± 0.071 | 0.077 ± 0.071 | 0.083 ± 0.072 |
+| global_ridge | 0.126 ± 0.128 | 0.137 ± 0.126 | 0.135 ± 0.104 |
+| topk8_global_ridge | 0.117 ± 0.109 | 0.124 ± 0.101 | 0.121 ± 0.124 |
+| pertensor_ridge | 0.115 ± 0.109 | 0.121 ± 0.093 | 0.126 ± 0.109 |
+| procrustes | 0.105 ± 0.082 | 0.104 ± 0.086 | 0.100 ± 0.074 |
+| pertensor_pca | 0.100 ± 0.076 | 0.100 ± 0.080 | 0.114 ± 0.103 |
+
+## 1C. Per-task paper table
+
+| Domain | Task | base_Y | mean_N16 | best_R6_N16 | best_R8_new_N16 | best_learned_N16 | oracle | gap_recovered | winner |
+|---|---|---|---|---|---|---|---|---|---|
+| math | gsm_hard | 0.063 | 0.066±0.005 | 0.066±0.005 (mean) | 0.070±0.007 (pertensor_pca) | 0.070±0.007 (pertensor_pca) | 0.150 | 0.077±0.077 | R8:pertensor_pca |
+| math | gsm8k_test_500 | 0.080 | 0.102±0.002 | 0.102±0.002 (mean) | 0.106±0.002 (pertensor_pca) | 0.106±0.002 (pertensor_pca) | 0.293 | 0.120±0.009 | R8:pertensor_pca |
+| code | mbpp_test_held | 0.230 | 0.240±0.000 | 0.257±0.006 (global_ridge) | 0.250±0.000 (pertensor_ridge) | 0.257±0.006 (global_ridge) | 0.320 | 0.296±0.064 | R6:global_ridge |
+| code | mbpp_plus | 0.217 | 0.212±0.002 | 0.270±0.003 (global_ridge) | 0.266±0.002 (pertensor_ridge) | 0.270±0.003 (global_ridge) | 0.450 | 0.229±0.014 | R6:global_ridge |
+| science | openbookqa_test | 0.710 | 0.754±0.002 | 0.754±0.002 (mean) | 0.756±0.019 (pertensor_pca) | 0.756±0.019 (pertensor_pca) | 0.983 | 0.167±0.070 | R8:pertensor_pca |
+
+
+## Headline
+
+Best aggregate (N, method) cell across R6 ∪ R8: R6 N=16 method=global_ridge gap_recovered=0.137 ± 0.126.
+Best individual task/seed cell: R6 task=mbpp_test_held N=4 seed=1 method=global_ridge gap_recovered=0.556 acc=0.280.
+Per-task winners over all evaluated R6 ∪ R8 cells: gsm_hard: R6:topk8_global_ridge N=8 seed=0; gsm8k_test_500: R6:mean N=12 seed=1; mbpp_test_held: R6:global_ridge N=4 seed=1; mbpp_plus: R6:topk8_global_ridge N=24 seed=0; openbookqa_test: R8:pertensor_pca N=12 seed=2.
+
+## Honest failure modes
+
+- No surrogate was used: every R8 accuracy comes from the same `model.generate(do_sample=False, num_beams=1)` path as R6.
+- Degenerate cells (gap_recovered ≤ -0.25): gsm_hard:topk2_global_ridge:N24:seed0=-0.31
+- Cross-domain/locality failures at K<5: none on aggregate, though individual tasks vary.
+- Seed instability (std > 0.15): none for 1B aggregate method×N cells.
+- Code-task evaluation remains the R4/R6 cheap string/span proxy rather than sandboxed unit tests.