--- license: apache-2.0 base_model: jakeatx/slimder-qwen38-ream288-depth32-agentic-ngram50-mixed-nvfp4-v2 pipeline_tag: text-generation tags: - peft - lora - qlora - qwen - mixture-of-experts - ream - nvfp4 - fp8 --- # REAM-288 Mixed NVFP4/FP8 Recovery LoRA v2 This repository contains the promoted recovery adapter for `jakeatx/slimder-qwen38-ream288-depth32-agentic-ngram50-mixed-nvfp4-v2`. The root adapter is the 50k-token checkpoint blended at 62.5% of its trained LoRA strength (`r=8`, trained `alpha=16`, promoted `alpha=10`). This was the strongest tested blend that passed every sealed behavior-preservation gate. ## Promoted adapter results - Recovery tokens: 50,176 non-benchmark tokens - Trainable parameters: 10,158,080 - Baseline sealed loss: 4.258437 - Promoted sealed loss: 3.958144 (improvement 0.300293) - Token KL: 0.176316 (maximum 0.20) - Router top-10 agreement: 0.924538 (minimum 0.90) - PLE cosine: 1.000004 (minimum 0.98) - Qualitative catastrophic failures: 0 - Gate result: **passed** The 250k training milestone also improved held-out validation loss from 1.773725 at 50k to 1.596309, but it was not promoted: its sealed token KL was 1.002711, router agreement was 0.834931, and it produced one Japanese character-loop failure. The full-strength 50k adapter was also not promoted because token KL was 0.329827. These checkpoints and their evidence are kept under `research-checkpoints/` for reproducibility, not as recommended defaults. ## Training - Sequence length: 512 - Learning rate: 2e-5 - LoRA rank: 8 - Trained LoRA alpha: 16 - Promoted LoRA alpha: 10 - Training corpus: 500 recovery rows plus 10 held-out validation rows - AAII, Frontier, AgentX, and other evaluation questions were excluded ## Usage Load the mixed v2 base checkpoint with its included custom runtime, then apply the adapter in this repository with PEFT. The base uses experimental `qwen4_exp`, compact PLE remapping, NVFP4 routed experts, and rowwise FP8 PLE storage, so it is not a generic drop-in Transformers checkpoint. ## Evidence `evidence/` contains the sealed comparison reports, training state, runtime validation, precision audits, and hashes. The strengthened qualitative checker detects both whitespace-token repetition and repeated non-ASCII character loops. ## Limitations This is an experimental recovery adapter. It passed the bounded sealed gate, but broad task and safety evaluation are still required before production use. ## Qwen3.8 Perian project lineage This repository is retained in the [Qwen3.8 Perian checkpoints collection](https://huggingface.co/collections/jakeatx/qwen38-perian-checkpoints-6aa33bb7acf0eba80edece40). Its exact position in the lineage is: **Early bounded recovery-QLoRA pilot on the fully compressed mixed base; it predates the final post-training run.** The final [Qwen3.8 Perian GGUF release](https://huggingface.co/jakeatx/Qwen3.8-Perian-GGUF) combines three reductions and one post-training stage: - depth: **48 to 32 transformer layers**; - routed-expert width: **384 to 288 experts per layer**; - PLE n-gram capacity: **320,001,446 to 160,000,768 rows** (50%, about 25.60B parameters removed), using activation-aware bigram and frequency-ranked trigram selections validated on a document-disjoint 5M-token holdout; - **rank-32 QLoRA** on **12,558 normalized traces** spanning math/STEM reasoning, coding/debugging, agentic tool use, retrieval, and general multi-step reasoning. The trace mixture draws from several frontier-model families, including Fable 5, GLM 5.2, Kimi K3, Claude Opus 4.7, Qwen3.8-Max, and GPT-5.6-Sol. The final merged milestone was trained through **9,336,692 supervised assistant tokens**. Earlier checkpoints in this collection do not inherit later stages merely by being listed beside them; the stage statement above is authoritative for this artifact.