File size: 2,108 Bytes
c2b6246 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
license: apache-2.0
tags:
- low-rank-fast-weights
- linear-attention
- gated_deltanet
- r512
- 350M
datasets:
- HuggingFaceFW/fineweb-edu
---
# GatedDeltaNet 350M (rank 512) — Low-rank Fast-Weight Ablation
Pretrained 350M-parameter GatedDeltaNet with low-rank parameterization
(`r512`) on FineWeb-Edu. Part of a multi-cell ablation across
4 archs × {`r32`, `r64`, `r256`, `rfull`} (plus GDN extras `r512`)
studying whether constraining the q/k/v fast-weight projections (or LaCT's
SwiGLU MLP) to low rank can match or exceed full-rank performance at the
350M scale.
## Training
| | |
|---|---|
| Architecture | GatedDeltaNet |
| Rank | `r512` |
| Params | ~350M (hidden=1024, layers=24, heads=16) |
| Dataset | `HuggingFaceFW/fineweb-edu` (streaming) |
| Steps | 10000 |
| Effective batch | 256 |
| Sequence length | 8000 |
| Optimizer | AdamW (lr=3e-4, eps=1e-15) |
| LR schedule | Cosine, 512-step warmup, decay to 10% |
| Precision | bf16 |
| Activation checkpointing | selective (option 1) |
| Tokens | ~20.5 B |
## Eval results
- **FineWeb-Edu val PPL**: `12.47`
- **LAMBADA acc**: 0.304
- **HellaSwag acc_norm**: 0.394
- **ARC-Easy acc_norm**: 0.478
- **ARC-Challenge acc_norm**: 0.273
- **PIQA acc_norm**: 0.663
- **WinoGrande acc**: 0.519
## Notes on the 350M sweep
- **Downstream eval discrimination comes online at 350M.** At 100M, HellaSwag /
LAMBADA were near-chance for most cells; at 350M they discriminate clearly
between archs/ranks.
- **PPL doesn't linearly predict downstream.** At matched ~374M, GLA `rfull`
has worse FineWeb-Edu PPL than DeltaNet `rfull` (14.42 vs 12.55) but wins on
*every* lm-harness task (LAMBADA, HellaSwag, PIQA, ARC-E).
- **GatedDeltaNet dominates at the cost of size.** GDN `rfull` is ~526M
(head_dim=256 inflates q/k/v) and wins every metric; GDN `r256` (~432M) is
the matched-param comparison and still leads.
- **LaCT is rank-robust at 350M.** PPL/LAMBADA stay flat across r64 / r256 /
rfull — the cleanest evidence for the "low rank as regularization"
hypothesis.
Run name: `gated_deltanet_350M_r512_bs256_lr3e-4_steps10000`
|