# 5.7M MC × Fractal RoPE ablation

Four small trains. Same FineWeb-Edu blend, same 10k steps, same looped
stack. Only two flags change. This is **not** Byrne-100M-Ultra-MC. It is
the isolation run that 114M card could not do.

| directory | MC | RoPE | params | Wiki @1024 seed 1 | Wiki @1024 seed 2 |
|---|---|---|---:|---:|---:|
| `std/` | off | geometric | 5.70M | 34.82 | **30.86** |
| `mc/` | on | geometric | 5.85M | 41.23 | — |
| `fractal/` | off | γ=1 | 5.70M | 40.49 | — |
| `both/` | on | γ=1 | 5.85M | **32.82** | 33.32 |

These folders are **seed 1**. Seed 2 retrained `std` and `both` only
(not uploaded here). Eval repeat matched to the printed digits. The
seed gap is training, not measurement.

Each folder is `config.json` + `model.safetensors` at step 10k.
Shared tokenizer is `tokenizer.json` in this directory (same 16,512
Length-MAX tokenizer as the 114M model; Dong & Su, arXiv:2511.20849).

Load with the SpikeWhale code in the parent repo. These nets are
hidden 128, 6 layers, loop ×3, seq 1024, `mc_segment_len` 256.

## What this is allowed to say

**MC alone did not help** at this size. Worse PPL, worse stretch to
2048/4096.

**Fractal alone did not help.** Worse PPL, worse stretch.

**Putting fractal RoPE on a finished `std` net after training** (nothing
else on) did not help. Unique-text 1024: 46.75 → 54.95.

**Both together** beat `std` at train length on seed 1 (Wiki 32.82 vs
34.82; unique 26.67 vs 46.75). On seed 2, `std` won Wiki (30.86 vs
33.32) and unique (13.35 vs 32.03). That 1024 win did not replicate.
Past 1024, `both` still fell off faster than `std` on both seeds.
Stretching the window favoured plain geometric.

MQAR 0/16 on every cell, both seeds.

## Why `both`, maybe

Seed 1 looked like an interaction. Seed 2 did not copy the Wiki win.
Gate *strength* on seed 1 is similar (`mean |tanh|` ~0.25 on `mc` vs
~0.26 on `both`). Sign is not: `mc` mixed +/−, layer 0 never left
zero; `both` layers 1–4 the same sign. Knock-out: `both` seed 1
26.67 → 33.93 unique PPL when the gates are zeroed; seed 2 32.03 →
42.45; `mc` 161 → 170 on an already-bad net. Hypothesis from seed 1:
at this width the branch settled next to Cantor frequencies and
fought itself next to geometric. It does not explain std unique PPL
moving 46.75 → 13.35 across seeds. I would not call `both` the
winner at 5M.

## 10k steps and “growing” memory

The paper’s *Growing Memory* (arXiv:2602.24281) is **not** “the branch
gets stronger the longer you train.” It is: cache a memory checkpoint
per **segment of the sequence**, so effective memory can grow with
**context length** (more segments → more cached states, cost between
O(L) and O(L²)).

What *does* grow with training **in this implementation** is the
zero-init mix-in `attn_out += tanh(mc_gate) * mc_out`. At step 0 that
is a no-op. On the 114M run, knock-out went from ~9% PPL early to
−16.7% on DPO, and mean |tanh(gate)| reached ~0.60. On these 5.7M
nets at 10k, |tanh| is ~0.25. So 10k may be early for the mix-in to
finish opening, especially the `mc`-only cell whose gates never
aligned. That is from these weights, not a claim in the paper. A
longer 5M train could still change the table. I have not run one.

## Files

```
5m-ablation/
  README.md
  tokenizer.json
  std/       config.json  model.safetensors
  mc/        config.json  model.safetensors
  fractal/   config.json  model.safetensors
  both/      config.json  model.safetensors
```