| --- |
| license: apache-2.0 |
| tags: |
| - low-rank-fast-weights |
| - linear-attention |
| - gated_deltanet |
| - r32 |
| - 100M |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| --- |
| |
| # GatedDeltaNet 100M (rank 32) — Low-rank Fast-Weight Ablation |
|
|
| Pretrained 100M-parameter GatedDeltaNet with low-rank parameterization |
| (`r32`) on FineWeb-Edu. Part of a 16-cell ablation |
| (4 archs × 4 ranks: `r8`, `r32`, `r64`, `rfull`) studying whether constraining |
| the q/k/v fast-weight projections (or LaCT's SwiGLU MLP) to low rank can match |
| or exceed full-rank performance. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Architecture | GatedDeltaNet | |
| | Rank | `r32` | |
| | Params | ~100M | |
| | Dataset | `HuggingFaceFW/fineweb-edu` (streaming) | |
| | Steps | 5000 | |
| | Effective batch | 256 | |
| | Sequence length | 8000 | |
| | Optimizer | AdamW (lr=3e-4, eps=1e-15) | |
| | LR schedule | Cosine, 256-step warmup, decay to 10% | |
| | Precision | bf16 | |
| | Activation checkpointing | selective (option 1) | |
| | Tokens | ~10.24 B | |
|
|
| Code: see [run_main_100M.sh](https://github.com/<your-repo>/lact_llm/lowrank_experiment). |
|
|
| ## Eval results |
|
|
| - **FineWeb-Edu val PPL**: `21.95` |
| - **MQAR (multi-query associative recall)**: |
| - `K=4`: 0.477 |
| - `K=16`: 0.806 |
| - `K=64`: 0.947 |
| - `K=256`: 0.986 |
| - **LAMBADA acc**: 0.111 |
| - **HellaSwag acc_norm**: 0.288 |
| - **ARC-Easy acc_norm**: 0.403 |
| - **PIQA acc_norm**: 0.591 |
| - **WinoGrande acc**: 0.486 |
| |
| |
| ## Notes |
| |
| - This is one of 16 cells; the other rank/arch combinations are uploaded under the |
| same HF org (`nlproj`) with repo names matching the local dump folder, e.g. |
| `nlproj/gated_deltanet_100M_{r8|r32|r64|rfull}_bs256_lr3e-4_steps5000`. |
| - **Key finding** of the ablation: at this scale, low rank often *matches or beats* |
| full rank on downstream tasks (LoRA-style "adaptation is intrinsically low-rank" |
| hypothesis). GatedDeltaNet is the exception — its rfull is the strongest in the |
| whole sweep on PPL / LAMBADA / HellaSwag / ARC-Easy. |
| |
| Run name: `gated_deltanet_100M_r32_bs256_lr3e-4_steps5000` |
| |
