| --- |
| license: apache-2.0 |
| tags: |
| - low-rank-fast-weights |
| - linear-attention |
| - gated_deltanet |
| - r512 |
| - 350M |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| --- |
| |
| # GatedDeltaNet 350M (rank 512) — Low-rank Fast-Weight Ablation |
|
|
| Pretrained 350M-parameter GatedDeltaNet with low-rank parameterization |
| (`r512`) on FineWeb-Edu. Part of a multi-cell ablation across |
| 4 archs × {`r32`, `r64`, `r256`, `rfull`} (plus GDN extras `r512`) |
| studying whether constraining the q/k/v fast-weight projections (or LaCT's |
| SwiGLU MLP) to low rank can match or exceed full-rank performance at the |
| 350M scale. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Architecture | GatedDeltaNet | |
| | Rank | `r512` | |
| | Params | ~350M (hidden=1024, layers=24, heads=16) | |
| | Dataset | `HuggingFaceFW/fineweb-edu` (streaming) | |
| | Steps | 10000 | |
| | Effective batch | 256 | |
| | Sequence length | 8000 | |
| | Optimizer | AdamW (lr=3e-4, eps=1e-15) | |
| | LR schedule | Cosine, 512-step warmup, decay to 10% | |
| | Precision | bf16 | |
| | Activation checkpointing | selective (option 1) | |
| | Tokens | ~20.5 B | |
|
|
| ## Eval results |
|
|
| - **FineWeb-Edu val PPL**: `12.47` |
| - **LAMBADA acc**: 0.304 |
| - **HellaSwag acc_norm**: 0.394 |
| - **ARC-Easy acc_norm**: 0.478 |
| - **ARC-Challenge acc_norm**: 0.273 |
| - **PIQA acc_norm**: 0.663 |
| - **WinoGrande acc**: 0.519 |
|
|
|
|
| ## Notes on the 350M sweep |
|
|
| - **Downstream eval discrimination comes online at 350M.** At 100M, HellaSwag / |
| LAMBADA were near-chance for most cells; at 350M they discriminate clearly |
| between archs/ranks. |
| - **PPL doesn't linearly predict downstream.** At matched ~374M, GLA `rfull` |
| has worse FineWeb-Edu PPL than DeltaNet `rfull` (14.42 vs 12.55) but wins on |
| *every* lm-harness task (LAMBADA, HellaSwag, PIQA, ARC-E). |
| - **GatedDeltaNet dominates at the cost of size.** GDN `rfull` is ~526M |
| (head_dim=256 inflates q/k/v) and wins every metric; GDN `r256` (~432M) is |
| the matched-param comparison and still leads. |
| - **LaCT is rank-robust at 350M.** PPL/LAMBADA stay flat across r64 / r256 / |
| rfull — the cleanest evidence for the "low rank as regularization" |
| hypothesis. |
| |
| Run name: `gated_deltanet_350M_r512_bs256_lr3e-4_steps10000` |
| |
