5.7M MC × Fractal RoPE ablation
Four small trains. Same FineWeb-Edu blend, same 10k steps, same looped stack. Only two flags change. This is not Byrne-100M-Ultra-MC. It is the isolation run that 114M card could not do.
| directory | MC | RoPE | params | Wiki @1024 seed 1 | Wiki @1024 seed 2 |
|---|---|---|---|---|---|
std/ |
off | geometric | 5.70M | 34.82 | 30.86 |
mc/ |
on | geometric | 5.85M | 41.23 | — |
fractal/ |
off | γ=1 | 5.70M | 40.49 | — |
both/ |
on | γ=1 | 5.85M | 32.82 | 33.32 |
These folders are seed 1. Seed 2 retrained std and both only
(not uploaded here). Eval repeat matched to the printed digits. The
seed gap is training, not measurement.
Each folder is config.json + model.safetensors at step 10k.
Shared tokenizer is tokenizer.json in this directory (same 16,512
Length-MAX tokenizer as the 114M model; Dong & Su, arXiv:2511.20849).
Load with the SpikeWhale code in the parent repo. These nets are
hidden 128, 6 layers, loop ×3, seq 1024, mc_segment_len 256.
What this is allowed to say
MC alone did not help at this size. Worse PPL, worse stretch to 2048/4096.
Fractal alone did not help. Worse PPL, worse stretch.
Putting fractal RoPE on a finished std net after training (nothing
else on) did not help. Unique-text 1024: 46.75 → 54.95.
Both together beat std at train length on seed 1 (Wiki 32.82 vs
34.82; unique 26.67 vs 46.75). On seed 2, std won Wiki (30.86 vs
33.32) and unique (13.35 vs 32.03). That 1024 win did not replicate.
Past 1024, both still fell off faster than std on both seeds.
Stretching the window favoured plain geometric.
MQAR 0/16 on every cell, both seeds.
Why both, maybe
Seed 1 looked like an interaction. Seed 2 did not copy the Wiki win.
Gate strength on seed 1 is similar (mean |tanh| ~0.25 on mc vs
~0.26 on both). Sign is not: mc mixed +/−, layer 0 never left
zero; both layers 1–4 the same sign. Knock-out: both seed 1
26.67 → 33.93 unique PPL when the gates are zeroed; seed 2 32.03 →
42.45; mc 161 → 170 on an already-bad net. Hypothesis from seed 1:
at this width the branch settled next to Cantor frequencies and
fought itself next to geometric. It does not explain std unique PPL
moving 46.75 → 13.35 across seeds. I would not call both the
winner at 5M.
10k steps and “growing” memory
The paper’s Growing Memory (arXiv:2602.24281) is not “the branch gets stronger the longer you train.” It is: cache a memory checkpoint per segment of the sequence, so effective memory can grow with context length (more segments → more cached states, cost between O(L) and O(L²)).
What does grow with training in this implementation is the
zero-init mix-in attn_out += tanh(mc_gate) * mc_out. At step 0 that
is a no-op. On the 114M run, knock-out went from ~9% PPL early to
−16.7% on DPO, and mean |tanh(gate)| reached ~0.60. On these 5.7M
nets at 10k, |tanh| is ~0.25. So 10k may be early for the mix-in to
finish opening, especially the mc-only cell whose gates never
aligned. That is from these weights, not a claim in the paper. A
longer 5M train could still change the table. I have not run one.
Files
5m-ablation/
README.md
tokenizer.json
std/ config.json model.safetensors
mc/ config.json model.safetensors
fractal/ config.json model.safetensors
both/ config.json model.safetensors