README.md

metadata

license: apache-2.0
tags:
  - low-rank-fast-weights
  - linear-attention
  - gated_deltanet
  - r512
  - 350M
datasets:
  - HuggingFaceFW/fineweb-edu

GatedDeltaNet 350M (rank 512) — Low-rank Fast-Weight Ablation

Pretrained 350M-parameter GatedDeltaNet with low-rank parameterization (r512) on FineWeb-Edu. Part of a multi-cell ablation across 4 archs × {r32, r64, r256, rfull} (plus GDN extras r512) studying whether constraining the q/k/v fast-weight projections (or LaCT's SwiGLU MLP) to low rank can match or exceed full-rank performance at the 350M scale.

Training

Architecture	GatedDeltaNet
Rank	r512
Params	~350M (hidden=1024, layers=24, heads=16)
Dataset	HuggingFaceFW/fineweb-edu (streaming)
Steps	10000
Effective batch	256
Sequence length	8000
Optimizer	AdamW (lr=3e-4, eps=1e-15)
LR schedule	Cosine, 512-step warmup, decay to 10%
Precision	bf16
Activation checkpointing	selective (option 1)
Tokens	~20.5 B

Eval results

• FineWeb-Edu val PPL: 12.47
• LAMBADA acc: 0.304
• HellaSwag acc_norm: 0.394
• ARC-Easy acc_norm: 0.478
• ARC-Challenge acc_norm: 0.273
• PIQA acc_norm: 0.663
• WinoGrande acc: 0.519

Notes on the 350M sweep

• Downstream eval discrimination comes online at 350M. At 100M, HellaSwag / LAMBADA were near-chance for most cells; at 350M they discriminate clearly between archs/ranks.
• PPL doesn't linearly predict downstream. At matched ~374M, GLA rfull has worse FineWeb-Edu PPL than DeltaNet rfull (14.42 vs 12.55) but wins on every lm-harness task (LAMBADA, HellaSwag, PIQA, ARC-E).
• GatedDeltaNet dominates at the cost of size. GDN rfull is 526M (head_dim=256 inflates q/k/v) and wins every metric; GDN r256 (432M) is the matched-param comparison and still leads.
• LaCT is rank-robust at 350M. PPL/LAMBADA stay flat across r64 / r256 / rfull — the cleanest evidence for the "low rank as regularization" hypothesis.

Run name: gated_deltanet_350M_r512_bs256_lr3e-4_steps10000