README.md

metadata

license: apache-2.0
tags:
  - low-rank-fast-weights
  - linear-attention
  - gated_deltanet
  - r8
  - 100M
datasets:
  - HuggingFaceFW/fineweb-edu

GatedDeltaNet 100M (rank 8) — Low-rank Fast-Weight Ablation

Pretrained 100M-parameter GatedDeltaNet with low-rank parameterization (r8) on FineWeb-Edu. Part of a 16-cell ablation (4 archs × 4 ranks: r8, r32, r64, rfull) studying whether constraining the q/k/v fast-weight projections (or LaCT's SwiGLU MLP) to low rank can match or exceed full-rank performance.

Training

Architecture	GatedDeltaNet
Rank	r8
Params	~100M
Dataset	HuggingFaceFW/fineweb-edu (streaming)
Steps	5000
Effective batch	256
Sequence length	8000
Optimizer	AdamW (lr=3e-4, eps=1e-15)
LR schedule	Cosine, 256-step warmup, decay to 10%
Precision	bf16
Activation checkpointing	selective (option 1)
Tokens	~10.24 B

Code: see run_main_100M.sh.

Eval results

• FineWeb-Edu val PPL: 26.31
• MQAR (multi-query associative recall):
  • K=4: 0.380
  • K=16: 0.799
  • K=64: 0.945
  • K=256: 0.986
• LAMBADA acc: 0.037
• HellaSwag acc_norm: 0.282
• ARC-Easy acc_norm: 0.402
• PIQA acc_norm: 0.594
• WinoGrande acc: 0.517

Notes

• This is one of 16 cells; the other rank/arch combinations are uploaded under the same HF org (nlproj) with repo names matching the local dump folder, e.g. nlproj/gated_deltanet_100M_{r8|r32|r64|rfull}_bs256_lr3e-4_steps5000.
• Key finding of the ablation: at this scale, low rank often matches or beats full rank on downstream tasks (LoRA-style "adaptation is intrinsically low-rank" hypothesis). GatedDeltaNet is the exception — its rfull is the strongest in the whole sweep on PPL / LAMBADA / HellaSwag / ARC-Easy.

Run name: gated_deltanet_100M_r8_bs256_lr3e-4_steps5000