README.md

---
license: apache-2.0
tags:
- low-rank-fast-weights
- linear-attention
- gated_deltanet
- r32
- 100M
datasets:
- HuggingFaceFW/fineweb-edu
---

# GatedDeltaNet 100M (rank 32) — Low-rank Fast-Weight Ablation

Pretrained 100M-parameter GatedDeltaNet with low-rank parameterization
(`r32`) on FineWeb-Edu. Part of a 16-cell ablation
(4 archs × 4 ranks: `r8`, `r32`, `r64`, `rfull`) studying whether constraining
the q/k/v fast-weight projections (or LaCT's SwiGLU MLP) to low rank can match
or exceed full-rank performance.

## Training

| | |
|---|---|
| Architecture | GatedDeltaNet |
| Rank | `r32` |
| Params | ~100M |
| Dataset | `HuggingFaceFW/fineweb-edu` (streaming) |
| Steps | 5000 |
| Effective batch | 256 |
| Sequence length | 8000 |
| Optimizer | AdamW (lr=3e-4, eps=1e-15) |
| LR schedule | Cosine, 256-step warmup, decay to 10% |
| Precision | bf16 |
| Activation checkpointing | selective (option 1) |
| Tokens | ~10.24 B |

Code: see [run_main_100M.sh](https://github.com/<your-repo>/lact_llm/lowrank_experiment).

## Eval results

- **FineWeb-Edu val PPL**: `21.95`
- **MQAR (multi-query associative recall)**:
  - `K=4`: 0.477
  - `K=16`: 0.806
  - `K=64`: 0.947
  - `K=256`: 0.986
- **LAMBADA acc**: 0.111
- **HellaSwag acc_norm**: 0.288
- **ARC-Easy acc_norm**: 0.403
- **PIQA acc_norm**: 0.591
- **WinoGrande acc**: 0.486


## Notes

- This is one of 16 cells; the other rank/arch combinations are uploaded under the
  same HF org (`nlproj`) with repo names matching the local dump folder, e.g.
  `nlproj/gated_deltanet_100M_{r8|r32|r64|rfull}_bs256_lr3e-4_steps5000`.
- **Key finding** of the ablation: at this scale, low rank often *matches or beats*
  full rank on downstream tasks (LoRA-style "adaptation is intrinsically low-rank"
  hypothesis). GatedDeltaNet is the exception — its rfull is the strongest in the
  whole sweep on PPL / LAMBADA / HellaSwag / ARC-Easy.

Run name: `gated_deltanet_100M_r32_bs256_lr3e-4_steps5000`