# Byrne-100M-Ultra-MC - Benchmark Scores

All scores for the released checkpoints (base pretrained, SFT, DPO). Same
byte-level tokenizer across all, so numbers are directly comparable.

_Generated 2026-08-18_

| checkpoint | stage | context |
|---|---|---|
| base-62k | pretrained (Dolma-blend, step 62k) | 1024 |
| sft-7100 | SFT on UltraChat (step 7.1k) | 4096 |
| dpo-3200 | DPO on the SFT model (step 3.2k) | 1024 |

---

## 1. Downstream benchmarks (lm-eval-harness style)

Scale: **capped** - 200 examples/MC-task, BLiMP 150/paradigm (12 paradigms), ArithMark 500.
Full-scale (uncapped) re-run in progress; this file will be updated with tighter stderr when it finishes. Binomial stderr = sqrt(p(1-p)/n).

| metric | base-62k | sft-7100 | dpo-3200 | chance |
|---|---|---|---|---|
| WikiText-2 byte_ppl (down) | 2.308 | 2.383 | 2.385 | - |
| BLiMP acc (up) | 0.811 | 0.780 | 0.779 | 0.50 |
| arc_easy acc | 0.410 | 0.395 | 0.400 | 0.25 |
| arc_easy acc_norm | 0.390 | 0.400 | 0.400 | 0.25 |
| arc_challenge acc | 0.205 | 0.265 | 0.255 | 0.25 |
| arc_challenge acc_norm | 0.270 | 0.260 | 0.260 | 0.25 |
| hellaswag acc | 0.370 | 0.375 | 0.370 | 0.25 |
| hellaswag acc_norm | 0.435 | 0.420 | 0.415 | 0.25 |
| winogrande acc | 0.515 | 0.500 | 0.505 | 0.50 |
| piqa acc | 0.565 | 0.590 | 0.590 | 0.50 |
| piqa acc_norm | 0.555 | 0.575 | 0.575 | 0.50 |
| openbookqa acc | 0.105 | 0.100 | 0.100 | 0.25 |
| openbookqa acc_norm | 0.290 | 0.285 | 0.295 | 0.25 |
| boolq acc | 0.355 | 0.430 | 0.425 | 0.50 |
| ArithMark-3.0 acc_norm | 0.354 | 0.380 | 0.378 | 0.25 |

Example counts (n): arc_easy=200, arc_challenge=200, hellaswag=200, winogrande=200, piqa=200, openbookqa=200, boolq=200, arithmark=500.

**Read:** base wins the pure-LM metrics (byte_ppl, BLiMP). SFT/DPO win the task
metrics - BoolQ, ARC-Challenge, ArithMark, PIQA - the instruction-tuning trade.
DPO is within noise of SFT here. Most MC tasks sit near chance at ~114M params.

---

## 2. Perplexity suite (per-token cross-entropy, lower is better)

10x1024-token streamed windows per source (WikiText-2: 40 windows).

| domain | base-62k | sft-7100 | dpo-3200 |
|---|---|---|---|
| Python-Edu (code) | 4.52 | 4.54 | 4.54 |
| Cosmopedia-v2 | 5.51 | 5.38 | 5.35 |
| FineMath-4+ | 6.32 | 6.84 | 6.86 |
| Dolma | 7.11 | 7.55 | 7.57 |
| Wikipedia | 7.26 | 7.83 | 7.84 |
| UltraX-web | 7.95 | 8.58 | 8.60 |
| DCLM | 8.21 | 9.10 | 9.12 |
| WikiText-2 | 9.08 | 10.11 | 10.13 |
| FineWeb-Edu | 10.53 | 11.69 | 11.70 |
| **mean (5-src)** | **6.50** | 6.89 | 6.89 |

mean (5-src) = Python-Edu, Cosmopedia, FineMath, UltraX, DCLM.

---

## 3. Memory Cache (base 62k + DPO 3.2k)

Paper tests from Behrouz et al. 2026 (arXiv:2602.24281) on `base_62k.pt`.
Same 6-trial S-NIAH grid and MQAR, MC on vs `mc_gate` zeroed. These
changed the direction: they are the recall evals, and this 114M base is
at chance either way. That is not proof MC helps, and not proof it does
not.

| test | MC on | MC off |
|---|---|---|
| MQAR n_kv=8 (20 queries) | 0/20 | 0/20 |
| MQAR n_kv=16 (20 queries) | 1/20 | 0/20 |
| S-NIAH-1 passkey (36 trials) | 0/36 | 0/36 |
| S-NIAH-2 numeric (36 trials) | 1/36 | 0/36 |

S-NIAH-2 MC-on hit is 1/6 at length 512 depth 0 only. n=6. Noise.

How it may be in the CE path (not a matched-train result):

| metric | value |
|---|---|
| gate strength DPO 3.2k (mean abs(tanh(gate)) over layers) | 0.599 (range -0.75 .. +0.90) |
| UltraX PPL, MC on | 9.87 |
| UltraX PPL, MC off (inference knock-out) | 11.86 |
| **knock-out, this checkpoint** | **-16.7% perplexity** |

Gates are open; deleting the branch at inference costs ~17% PPL (was ~9%
earlier in training). That is not a matched train. At 5.7M, training
without the branch (std) beat training with MC alone. See section 6.

---

## 4. Fractal RoPE - schedule dependence (base 62k, natural text)

Trained fractal (Cantor-spectrum, gamma=1) vs geometric RoPE swapped in
at inference on the same weights. In-distribution PPL. Dependence, not
"fractal is better."

| source | fractal (trained) | standard (swapped) | delta |
|---|---|---|---|
| UltraX-web | 7.95 | 77.86 | +879% |
| DCLM | 8.21 | 70.74 | +762% |
| FineWeb-Edu | 10.53 | 81.57 | +675% |
| WikiText-2 | 9.08 | 71.06 | +683% |

---

## 5. Fractal RoPE - both models, unique text, native vs swap

`eval_rope_swap.py`. Unique non-repeating records, 3 windows. Twin is
step 38.5k, MC off, geometric RoPE. Ultra-MC is step 62k, MC on, fractal.
Two knobs, unmatched steps. Native-vs-swap *within* a net is the clean
number. Cross-model 2048 is a hint, not an ablation.

| model | 256 | 512 | 1024 | 2048 |
|---|---:|---:|---:|---:|
| Ultra-MC 62k native (fractal) | 3.20 | 2.66 | 2.56 | 30.67 |
| Ultra-MC 62k swap → geometric | 3.22 | 3.55 | 21.65 | 141.90 |
| Twin 38.5k native (geometric) | 3.06 | 2.55 | 2.50 | 4.48 |
| Twin 38.5k swap → fractal γ=1 | 3.36 | 2.81 | 2.73 | 2.76 |

PPL by target position, 2048 window:

| pos | Ultra native | Ultra swap | Twin native | Twin swap |
|---|---:|---:|---:|---:|
| 0–256 | 3.78 | 3.79 | 3.64 | 3.93 |
| 256–512 | 2.21 | 3.94 | 2.20 | 2.37 |
| 512–768 | 2.28 | 127.32 | 2.25 | 2.43 |
| 768–1024 | 2.40 | 116.38 | 2.39 | 2.61 |
| 1024–1280 | 19.42 | 759.55 | 2.37 | 2.59 |
| 1280–1536 | 665 | 1132 | 2.68 | 2.57 |
| 1536–1792 | 1122 | 775 | 5.28 | 2.79 |
| 1792–2048 | 1171 | 1098 | 112.09 | 3.06 |

Ultra-MC: fractal load-bearing inside 1024; native also dies past 1024.
Twin: geometric load-bearing inside 1024; holds to ~1536; last 256 blows
on geometric and does not if you put fractal frequencies on those weights.

---

## 6. 5.7M matched 2x2 (10k steps, current FineWeb-Edu blend)

Isolation test the 114M twin is not. One seed. WikiText-2: 20 x 1024
windows, same tokenizer as the rest of this file.

| run | MC | RoPE | Wiki @1024 | unique @1024 | unique @2048 | unique @4096 | val_loss @10k |
|---|---|---|---:|---:|---:|---:|---:|
| std | off | geometric | 34.82 | 46.75 | 56.57 | 98.51 | 4.964 |
| mc | on | geometric | 41.23 | 161.49 | 199.41 | 267.31 | 5.055 |
| fractal | off | gamma=1 | 40.49 | 145.30 | 217.79 | 303.29 | 4.999 |
| both | on | gamma=1 | 32.82 | 26.67 | 76.41 | 158.95 | 4.867 |

Post-train RoPE switch on **std** (nothing else on): unique 1024
46.75 -> fractal-at-load 54.95; 2048 56.57 -> 60.58. Does not help.

MC knock-out (unique PPL): `both` 26.67 -> 33.93 at 1024; `mc` 161.49
-> 170.47. MQAR 0/16 all four.

**At this size:** MC alone no. Fractal alone no. Post-hoc fractal on
std no. Both-together yes at 1024, then worse stretch than std. Gate
*strength* similar on `mc` vs `both` (mean |tanh| ~0.25); `mc` gates
have mixed sign, `both` layers 1-4 the same sign. Hypothesis: the
branch only settled next to fractal frequencies. One seed. Not 114M.