mimo-7b-gdn-hybrid-stage1-align

Stage-1 endpoint — hidden-state alignment. Exit: rel-MSE 0.0077, wikitext PPL 52.9 (1.9x teacher). Accepted via KD-smoke functional gate (val KL 1.554->0.563 in 2 steps); raw-prose degeneracy at this stage is MiMo-specific calibration, fixed by KD.

Part of the MiMo second-model arm of a study converting full-attention models into GDN (gated DeltaNet) hybrids (uniform 1:4 retention) and recovering capability via staged distillation. Teacher/base: MiMo-7B-RL-0530.

NOTE: these checkpoints use the custom mimo_gdn architecture — register it before loading (import src.models.mimo_gdn from the project repo, or use the load_causal_lm helper in src/eval/sample_generations.py). Bare AutoModelForCausalLM will not work. vLLM serving needs src.models.vllm_mimo_gdn.register().

Pipeline: surgery/init (uniform 1:4 plain inheritance) -> stage-1 hidden-state alignment -> stage-2a forward-KL KD @4k -> stage-2b long-context KD @32k. Within-arm reference: teacher wikitext-2 PPL 27.51 (RL'd teacher; never cross-arm-compare absolute PPL).