first person; 5M 2x2 current; paper growing-memory vs zero-init gate

README.md

@@ -43,8 +43,10 @@ It is not on the generate path.
 | `checkpoints/base_62k.pt` | pretrained, Dolma-blend, step 62k | lowest PPL. Continues text, does not answer questions. |
 | `checkpoints/sft_7100.pt` | SFT on UltraChat, step 7.1k | chat / instruction following. `--chat` |
 | `checkpoints/dpo_3200.pt` | DPO on the SFT model, step 3.2k | preference-tuned chat (default). `--chat` |
+| [`5m-ablation/`](5m-ablation/README.md) | four ~5.7M 10k trains | MC × fractal isolation. Not this 114M model. |
 
-Same architecture, same loader. Weights and context length differ: SFT was
+Same architecture family, same loader for the 114M stages. The 5M
+nets are a separate, smaller stack. Weights and context length differ: SFT was
 trained at 4096, base and DPO at 1024.
 
 ## Architecture
@@ -165,13 +167,13 @@ bf16 and training blows up.
 
 This is not an RNN, so the paper's complexity story does not transfer cleanly.
 Cross-segment readout here is O(N²L). It is not a long-context speed trick.
-The thing that transferred is a second retrieval path with learned
-segment-level routing next to token-level attention, on a 114M model that
-does not have much attention to spare. Recall is the usual failure mode at
-this size. Zero-init gate means at step 0 the model is identical to the no-MC
-baseline; the branch only starts mattering if training uses it. Pretrain
-`seq_len` was 1024 = 4 segments, so it sees cross-segment memory from the
-start.
+The paper's "growing memory" is with **sequence length** (more segments →
+more cached states), not with training steps. What grows with training
+**here** is the zero-init mix-in `tanh(mc_gate)`: at step 0 the model is
+the no-MC baseline; the residual only starts mattering if that gate
+opens. On this 114M run it did (knock-out ~9% early → −16.7% at DPO;
+mean |tanh| ~0.60). Pretrain `seq_len` was 1024 = 4 segments, so the
+branch sees cross-segment memory from the first step.
 
 It did get used. DPO 3.2k checkpoint:
 
@@ -184,8 +186,10 @@ It did get used. DPO 3.2k checkpoint:
 
 Gates are open. Knocking the branch out costs ~17% PPL (it was ~9% earlier,
 before they opened). That is an inference knock-out on this checkpoint,
-not a model trained without MC. The twin in progress also dropped
-fractal RoPE, so it is not an MC-only ablation.
+not a model trained without MC. The 114M twin also dropped fractal RoPE,
+so it is not an MC-only ablation. A 5.7M 2×2 that *does* isolate the
+knobs is in [`5m-ablation/`](5m-ablation/README.md). MC alone did not
+help there. MC+fractal together did, at train length, one seed.
 
 If you write a loader: this branch is a full-sequence op. It uses the whole
 `x` given to `forward`. Naive KV cache feeds it one token, it rebuilds
@@ -259,11 +263,11 @@ What that actually says:
   does not extrapolate here; the geometric twin does; fractal
   frequencies on the twin also do. Matched 61k still required.
 
-### What we can and cannot claim about Memory Cache
+### What I can and cannot claim about Memory Cache
 
 Behrouz et al. 2026 (arXiv:2602.24281) score MC with matched RNN
 trains, then Wiki PPL, S-NIAH (passkey / numeric / UUID), in-context
-recall vs length, and MQAR. Those are the tests we ran on this 62k
+recall vs length, and MQAR. Those are the tests I ran on this 62k
 base, **MC on and with `mc_gate` zeroed**.
 
 They **changed the direction**. The earlier filler/needle story was not
@@ -325,10 +329,14 @@ Past 1024 it falls off faster than std (bin 1024–1280 already 101 vs
 std 46). Stretching the window is a win for plain geometric, not for
 MC. MQAR is 0/16 on every cell.
 
-That is a 5.7M / 10k result. It is not a 114M result.
+That is a 5.7M / 10k result. It is not a 114M result. 10k may also be
+early for the zero-init mix-in: |tanh(gate)| ~0.25 here vs ~0.60 on the
+114M DPO. The paper's growing memory is with **context length**, not
+steps. Gate-opening over training is this code. A longer 5M run is
+untested. Weights: [`5m-ablation/`](5m-ablation/README.md).
 
-**Why both, if each knob alone is worse.** We do not have a second
-seed, so this could still be luck. What we *did* measure:
+**Why both, if each knob alone is worse.** I do not have a second
+seed, so this could still be luck. What I measured:
 
 - `both` actually uses the branch. Unique PPL 26.67 → 33.93 when
   `mc_gate` is zeroed. `mc` alone is 161 → 170 on an already-bad net.
@@ -529,6 +537,7 @@ spike_infer/          engine
 engine_tools/         export, GGUF, long-context, multiturn
 morpho/               int8 circuit of one block (not used at generate time)
 GGUF/                 f16 + q8_0 for each stage
+5m-ablation/          four ~5.7M 10k trains (MC × fractal)
 ```
 
 ## Limitations
@@ -566,9 +575,11 @@ parts that were the actual work. Leave the chat stages.
 Worth copying:
 
 - The Memory Cache branch as a gated residual next to MLA, zero-init so
-  it can't hurt at step 0. Ablation is in the architecture section
-  (~17% PPL). Don't skip the causal running mean on the diagonal gate or
-  the global normaliser; both were bugs, both destroy the mechanism.
+  it can't hurt at step 0. The ~17% figure in the architecture section
+  is an inference knock-out on *this* 114M checkpoint, not a matched
+  train. At 5.7M, MC alone did not beat the no-MC cell. Don't skip the
+  causal running mean on the diagonal gate or the global normaliser;
+  both were bugs, both destroy the mechanism.
   Builders: see [For other model builders](#for-other-model-builders).
 - Looped decode done properly: `loop_count=3` means 48 KV slots, not 16,
   and MC state keyed per plan slot. A loader that indexes by layer only
@@ -608,11 +619,16 @@ below). I have not proven it helps at 114M. Paper recall tests on this
 62k base are chance with the branch on or off. A matched 5.7M 2×2 on
 the current blend says **MC alone did not help PPL or long context**;
 **MC+fractal together** was the only cell that beat the nothing-on
-baseline at train length. That is 5M / 10k, one seed. Do not promote
-it to this 114M card. The paper is about RNNs; this is not an RNN;
-the O(L) claim does not transfer.
+baseline at train length. That is 5M / 10k, one seed, and |tanh(gate)|
+there is ~0.25 vs ~0.60 on this 114M DPO — the mix-in may still be
+opening. The paper does not say "train longer and MC appears"; it says
+memory grows with **how many segments the sequence has**. The
+zero-init gate opening over steps is this implementation. A longer 5M
+run could still move that table. I have not run one. Do not promote
+the 10k result to this 114M card. The paper is about RNNs; this is not
+an RNN; the O(L) claim does not transfer.
 
-What we actually ran on *this* model, not what I hope it does:
+What I actually ran on *this* model, not what I hope it does:
 
 **Did training use it.** Yes. Mean |tanh(gate)| across layers is 0.599
 on the DPO checkpoint. Zeroing the branch at inference (same weights,
@@ -624,7 +640,7 @@ trained without the branch.
 256`, so cross-segment memory is in the recipe from step 0. SFT grew
 the window to 4096. Most of the stack is position-local. MC is not —
 behaviour only changes when a sequence crosses a 256-token boundary.
-We widened the config (`export_weights.py --seq-len 16384`, RoPE
+I widened the config (`export_weights.py --seq-len 16384`, RoPE
 caches rebuilt, no retrain of positions) and checked that cached
 decode still *is* the trained function:
 
@@ -646,17 +662,18 @@ is whether *it* collapses out there. Needle is a different question
 (the first needle was the model continuing the start of the prompt;
 see that section).
 
-**What is still missing.** A train that changes only MC, fractal left
-on. The twin that is running also dropped fractal RoPE, so it is not
-that experiment. It already informs RoPE (native vs swap CE and
-position bins on both nets, above). Paper recall tests (S-NIAH, MQAR)
-on this 62k base are at chance with MC on or off; they do not prove
-MC. If you copy the branch anyway, copy the two bugfixes or you will
-not have the mechanism: causal running mean on the diagonal gate, and
-one global normaliser (not per-segment). fp32, autocast off. State
-keyed per segment *and* per loop pass if you loop. Set `seq_len` to a
-multiple of `mc_segment_len` so the branch sees more than one segment
-from the first step.
+**What is still missing at 114M.** A train that changes only MC,
+fractal left on. The nomc twin also dropped fractal RoPE, so it is not
+that experiment. The 5.7M 2×2 *is* that experiment at small scale:
+MC alone did not help; MC+fractal together did, at 1024, one seed.
+Weights and write-up: [`5m-ablation/`](5m-ablation/README.md). Paper
+recall tests (S-NIAH, MQAR) on this 62k base are at chance with MC on
+or off; they do not prove MC. If you copy the branch anyway, copy the
+two bugfixes or you will not have the mechanism: causal running mean
+on the diagonal gate, and one global normaliser (not per-segment).
+fp32, autocast off. State keyed per segment *and* per loop pass if you
+loop. Set `seq_len` to a multiple of `mc_segment_len` so the branch
+sees more than one segment from the first step.
 
 ## Provenance