File size: 3,067 Bytes
d1e91f3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | # Round 8 — Fill R6 Gaps
R8 fills three gaps R6 left open: K-sweep at N=24, per-tensor and Procrustes methods, and a per-task breakdown.
## 1A. K-sweep at N=24
| K | aggregate gap_recovered ± std |
|---:|---:|
| 2 | 0.032 ± 0.251 |
| 4 | 0.069 ± 0.210 |
| 6 | 0.083 ± 0.136 |
| 8 | 0.129 ± 0.156 |
| 12 | 0.131 ± 0.118 |
| 16 | 0.129 ± 0.097 |
| 20 | 0.130 ± 0.089 |
| 24 | 0.126 ± 0.089 |
No K beats R6 global_ridge at N=24 (0.135); K=12 is best among Top-K rows at 0.131.
## 1B. Per-tensor methods
| method | N=12 | N=16 | N=24 |
|---|---:|---:|---:|
| mean | 0.077 ± 0.071 | 0.077 ± 0.071 | 0.083 ± 0.072 |
| global_ridge | 0.126 ± 0.128 | 0.137 ± 0.126 | 0.135 ± 0.104 |
| topk8_global_ridge | 0.117 ± 0.109 | 0.124 ± 0.101 | 0.121 ± 0.124 |
| pertensor_ridge | 0.115 ± 0.109 | 0.121 ± 0.093 | 0.126 ± 0.109 |
| procrustes | 0.105 ± 0.082 | 0.104 ± 0.086 | 0.100 ± 0.074 |
| pertensor_pca | 0.100 ± 0.076 | 0.100 ± 0.080 | 0.114 ± 0.103 |
## 1C. Per-task paper table
| Domain | Task | base_Y | mean_N16 | best_R6_N16 | best_R8_new_N16 | best_learned_N16 | oracle | gap_recovered | winner |
|---|---|---|---|---|---|---|---|---|---|
| math | gsm_hard | 0.063 | 0.066±0.005 | 0.066±0.005 (mean) | 0.070±0.007 (pertensor_pca) | 0.070±0.007 (pertensor_pca) | 0.150 | 0.077±0.077 | R8:pertensor_pca |
| math | gsm8k_test_500 | 0.080 | 0.102±0.002 | 0.102±0.002 (mean) | 0.106±0.002 (pertensor_pca) | 0.106±0.002 (pertensor_pca) | 0.293 | 0.120±0.009 | R8:pertensor_pca |
| code | mbpp_test_held | 0.230 | 0.240±0.000 | 0.257±0.006 (global_ridge) | 0.250±0.000 (pertensor_ridge) | 0.257±0.006 (global_ridge) | 0.320 | 0.296±0.064 | R6:global_ridge |
| code | mbpp_plus | 0.217 | 0.212±0.002 | 0.270±0.003 (global_ridge) | 0.266±0.002 (pertensor_ridge) | 0.270±0.003 (global_ridge) | 0.450 | 0.229±0.014 | R6:global_ridge |
| science | openbookqa_test | 0.710 | 0.754±0.002 | 0.754±0.002 (mean) | 0.756±0.019 (pertensor_pca) | 0.756±0.019 (pertensor_pca) | 0.983 | 0.167±0.070 | R8:pertensor_pca |
## Headline
Best aggregate (N, method) cell across R6 ∪ R8: R6 N=16 method=global_ridge gap_recovered=0.137 ± 0.126.
Best individual task/seed cell: R6 task=mbpp_test_held N=4 seed=1 method=global_ridge gap_recovered=0.556 acc=0.280.
Per-task winners over all evaluated R6 ∪ R8 cells: gsm_hard: R6:topk8_global_ridge N=8 seed=0; gsm8k_test_500: R6:mean N=12 seed=1; mbpp_test_held: R6:global_ridge N=4 seed=1; mbpp_plus: R6:topk8_global_ridge N=24 seed=0; openbookqa_test: R8:pertensor_pca N=12 seed=2.
## Honest failure modes
- No surrogate was used: every R8 accuracy comes from the same `model.generate(do_sample=False, num_beams=1)` path as R6.
- Degenerate cells (gap_recovered ≤ -0.25): gsm_hard:topk2_global_ridge:N24:seed0=-0.31
- Cross-domain/locality failures at K<5: none on aggregate, though individual tasks vary.
- Seed instability (std > 0.15): none for 1B aggregate method×N cells.
- Code-task evaluation remains the R4/R6 cheap string/span proxy rather than sandboxed unit tests.
|