qwen3-4b-gdn-hybrid-opd

Main result — stage-3 OPD endpoint (v2, clean token-keyed schedule). MATH-500 think 0.792, AIME'24/'25 avg@8 0.183/0.175 (pass@8 0.400/0.267), RULER worst cell 0.960 (vt@32k), GSM8K@1024 0.684, 0-shot avg 0.599, wikitext-2 PPL 15.65. Decode: ~flat tok/s vs length; 1.35x teacher at 32k x batch-8 with ~3.9x KV-cache capacity.

Part of a study converting full-attention Qwen3-4B into a GDN (gated DeltaNet) hybrid (27 of 36 layers converted, uniform 1:4 retention) and recovering capability via staged distillation, culminating in on-policy distillation (OPD): student rollouts scored per-token by the frozen teacher under reverse KL. Checkpoints are stock Qwen3NextForCausalLM — load with AutoModelForCausalLM (transformers >= 4.57). vLLM serving needs a small registry shim (all-dense MoE + tied-embeddings fixes) from the project repo.

Pipeline: surgery/init -> stage-1 hidden-state alignment (100M tok) -> stage-2a forward-KL KD @4k (600M) -> stage-2b long-context KD @32k (294M) -> stage-3 OPD (200M gen tokens, horizon 512->16k, token-keyed LR). Teacher/reference: Qwen/Qwen3-4B (thinking mode).