Rewrite README: minimal style, drop perf data, match nvidia M2.5 recipe details

README.md

@@ -1,187 +1,56 @@
 ---
 base_model:
 - MiniMaxAI/MiniMax-M2.7
-base_model_relation: quantized
 license: mit
-pipeline_tag: text-generation
-library_name: vllm
-tags:
-- nvfp4
-- fp4
-- quantization
-- modelopt
-- tensorrt-model-optimizer
-- mixture-of-experts
-- moe
-- minimax
 ---
 
-# MiniMax-M2.7 — NVFP4 (experts-only) + FP8 KV cache
-
-NVFP4 quantization of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) produced with
-[NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) (`modelopt` 0.43.0rc3).
-
-- **Base model:** MiniMax-M2.7 (230B total / 10B active, 256-expert MoE, 62 layers)
-- **Weight quant:** NVFP4 (W4A4, group size 16) — MoE expert MLPs only (`w1`, `w2`, `w3`)
-- **Attention / router / lm_head:** BF16 (unquantized)
-- **KV cache:** FP8 E4M3 with **static baked scales** (all `k_scale`/`v_scale` = 1.0 after calibration)
-- **Target hardware:** NVIDIA Blackwell (SM100 B200, SM120 RTX Pro 6000) with native NVFP4 tensor cores
-- **On-disk size:** 139.89 GB across 29 safetensors shards (vs. ~460 GB BF16)
-
-## What's quantized
-
-Only the expert MLP projections in each MoE block (`block_sparse_moe.experts.{0..255}.w{1,2,3}`)
-are converted to NVFP4. Everything else — self-attention (`q_proj`/`k_proj`/`v_proj`/`o_proj`),
-router (`block_sparse_moe.gate`), embeddings, layernorms, and `lm_head` — stays in BF16.
-This matches the `excluded_modules` declaration in [`hf_quant_config.json`](./hf_quant_config.json):
-
-```json
-{
-  "quantization": {
-    "quant_algo": "NVFP4",
-    "kv_cache_quant_algo": "FP8",
-    "group_size": 16,
-    "exclude_modules": [
-      "lm_head",
-      "*self_attn*",
-      "*block_sparse_moe.gate*"
-    ]
-  }
-}
-```
+## Model Description
 
-## Calibration recipe
-
-- **modelopt recipe:** `nvfp4_experts_only-fp8_kv` (variant of `nvfp4_default-fp8_kv`
-  that leaves attention projections in BF16)
-- **Calibration data:** 1024 samples — 512 from `cnn_dailymail` + 512 from
-  `nvidia/OpenMathReasoning` (math reasoning traces), each truncated to 2048 tokens
-- **Batch size:** 64, `seq_len=2048`
-- **All-expert activation:** `--moe_calib_experts_ratio 1.0` forces every forward pass
-  during calibration to dispatch to all 256 experts in each layer. This is critical —
-  natural top-k routing over 1024 samples leaves roughly 6 experts across the 62 layers
-  completely cold (amax=0), which in turn makes their NVFP4 block scales degenerate and
-  produces characteristic "spurious space" token glitches during inference (e.g.
-  `"mpg_ high"`, `"SPEC. md"`). Forcing full-expert coverage eliminates this class of
-  quality regression entirely.
-- **KV cache scales:** statically calibrated and baked into the safetensors (not computed
-  on-the-fly at serving time). This avoids the silent per-layer `k_scale`/`v_scale = 1.0`
-  fallback in vLLM when KV scales are missing from the checkpoint.
-- **Quantizer fix (modelopt 0.43.0rc3):** 124 `use_constant_amax` quantizers (self-attention
-  Q/K/V/O and router inputs) are missing `_amax` after calibration due to an upstream
-  modelopt bug. Before export we explicitly register `_amax = 448.0` on each one — the
-  correct FP8 E4M3 saturation value. Without this fix the export silently produces a
-  checkpoint that loads but generates garbage.
-- **Hardware:** 4x H100 PCIe (80 GB) — the base BF16 model streams through at ~9 GB/expert
-  layer; the run takes about 90 minutes end-to-end including calibration.
-
-## Quality
-
-Smoke-tested with vLLM v0.19.0 on 2x RTX Pro 6000 (SM120 Blackwell).
-
-- **Prose generation** (5-sentence summary of the Battle of Thermopylae): passes.
-  Correct dates (480 BCE), names (Leonidas, Xerxes, Ephialtes, Thespians, Salamis),
-  coherent narrative, no glitch tokens, `finish_reason=stop`.
-- **Tool calling** (single-turn `get_weather` function): passes. Correct function
-  selected, correct `{"location": "Boston"}` args, `<think>` reasoning properly
-  separated into `reasoning_content` via the `minimax_m2` parser.
-- **No spurious-space tokens** observed across subword-heavy prompts — the cold-expert
-  class of failure is gone.
-
-## Hardware requirements
-
-NVFP4 requires **native FP4 tensor cores**, available on:
-
-| GPU | SM | Status |
-|-----|-----|--------|
-| B200 / GB200 | SM100 | native |
-| B100 | SM100 | native |
-| RTX Pro 6000 (Blackwell) | SM120 | native |
-| DGX Spark (GB10) | SM121 | native (requires 128×128×128 CUTLASS tile) |
-| H100 / H200 | SM90 | **not supported** (no FP4) |
-| A100 | SM80 | **not supported** |
-
-Minimum VRAM for TP=2 at `--max-model-len=196608`: ~90 GB per GPU (FP8 KV cache).
-
-## Serving with vLLM
-
-Tested on **vLLM v0.19.0** with the FLASHINFER attention backend and `FLASHINFER_CUTLASS` NVFP4 MoE kernel (both auto-selected — no env vars needed).
-
-```bash
-python3 -m vllm.entrypoints.openai.api_server \
-  --model demon-zombie/MiniMax-M2.7-NVFP4 \
-  --trust-remote-code \
-  --served-model-name MiniMax-M2.7 \
-  --tensor-parallel-size 2 \
-  --max-model-len 196608 \
-  --max-num-seqs 256 \
-  --gpu-memory-utilization 0.90 \
-  --tool-call-parser minimax_m2 \
-  --reasoning-parser minimax_m2 \
-  --enable-auto-tool-choice \
-  --disable-custom-all-reduce \
-  --host 0.0.0.0 \
-  --port 8000
-```
+**MiniMax-M2.7-NVFP4** is an NVFP4-quantized version of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7), a 230B-parameter Mixture-of-Experts language model with 10B active parameters.
 
-### Required flags
+Produced with [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) (`modelopt` 0.43.0rc3) using the `nvfp4_experts_only-fp8_kv` recipe — the same recipe structure as [`nvidia/MiniMax-M2.5-NVFP4`](https://huggingface.co/nvidia/MiniMax-M2.5-NVFP4). The base model is quantized directly from its official FP8 checkpoint via modelopt's on-the-fly FP8→BF16 dequant path (no separate BF16 master), then requantized to NVFP4.
 
-- `--trust-remote-code` — the MiniMax M2 architecture uses custom Python modeling code.
-- `--disable-custom-all-reduce` — **critical on SM120 (RTX Pro 6000) with TP>1**.
-  vLLM's custom CUDA all-reduce hangs post-weight-load on Pro 6000: workers spin at
-  100% SM utilization with 0 memory traffic and minimal power draw, and EngineCore
-  logs `No available shared memory broadcast block found in 60s` every minute. Passing
-  `--disable-custom-all-reduce` falls back to NCCL and avoids the hang. This is required
-  even on tagged v0.19.0-cu130. Not needed on B200.
-- Do **not** pass `--enforce-eager`. Earlier reports suggested eager mode was required
-  on Blackwell — this is no longer true as of vLLM v0.19.0. CUDA graphs capture cleanly
-  (PIECEWISE mixed prefill-decode + FULL decode, 51 + 35 graphs captured) and give
-  roughly 5x single-stream throughput and 2x aggregate throughput at 8x concurrency
-  compared to eager. No accuracy regression.
+### What's quantized
 
-### Backend auto-selection
+Only the MoE expert MLP layers (`w1`, `w2`, `w3` — gate, up, down projections) are quantized to NVFP4 (W4A4, group size 16). Self-attention projections, the MoE router, embeddings, layernorms, and `lm_head` are left in BF16. The `exclude_modules` list in `hf_quant_config.json` matches `nvidia/MiniMax-M2.5-NVFP4` exactly.
 
-vLLM automatically picks:
+The KV cache is quantized to FP8 E4M3 with **statically calibrated scales** baked into the safetensors as per-layer `k_proj.k_scale` / `v_proj.v_scale` tensors (62 of each). This avoids vLLM's silent `scale=1.0` fallback when KV scales are missing from the checkpoint.
 
-- **NVFP4 MoE kernel:** `FLASHINFER_CUTLASS` (picked via `enable_sm120_or_later` logic)
-- **KV cache dtype:** `fp8_e4m3` (picked because `kv_cache_quant_algo: FP8` is in
-  `hf_quant_config.json` and the static `k_scale`/`v_scale` tensors are present in the shards)
-- **Attention backend:** `FLASHINFER`
+### Calibration
 
-## Performance
+- **Recipe:** `nvfp4_experts_only-fp8_kv`
+- **Dataset:** `cnn_dailymail` + `nvidia/Nemotron-Post-Training-Dataset-v2` (same as nvidia's M2.5 quant)
+- **Samples:** 1024
+- **Full expert coverage:** calibration is run with `--moe_calib_experts_ratio 1.0`, which sets the MoE top-k to 256 during calibration so every token activates every expert in every layer. This guarantees all 256 × 62 = 15,872 experts receive a non-zero `amax` regardless of dataset routing, and eliminates the cold-expert failure mode (spurious spaces at subword boundaries, e.g. `"mpg_ high"`, `"SPEC. md"`) seen when experts are calibrated via natural top-k routing alone.
 
-Measured on **2x RTX Pro 6000 (SM120 Blackwell, 96 GB each)**, vLLM v0.19.0, TP=2,
-`--disable-custom-all-reduce`, CUDA graphs enabled, `max-model-len=196608`:
+### Hardware
 
-### Sequential (single request, gen=500)
+NVFP4 requires native FP4 tensor cores — NVIDIA Blackwell (B100 / B200, RTX Pro 6000, DGX Spark GB10). Not supported on H100/H200 or older.
 
-| Context | tok/s | TTFT |
-|---------|------:|-----:|
-| warmup  |  80.7 | 1.26 s |
-| 9K      |  77.1 | 1.11 s |
-| 9K cached |  78.5 | 0.40 s |
-| 18K     |  76.0 | 1.25 s |
-| 27K     |  74.3 | 1.43 s |
-| 27K cached |  73.0 | 0.77 s |
+### Quality
 
-### Concurrent (ctx=32K, gen=500, 0 errors)
+Smoke-tested with vLLM v0.19.0 on 2x RTX Pro 6000 (SM120): prose generation and single-turn tool calling both pass, no spurious-space tokens observed. You should always evaluate against your specific use case.
 
-| Concurrency | agg tok/s | TTFT avg |
-|------------:|----------:|---------:|
-| 8x  | **201.8** | 6.95 s |
-| 16x | **427.8** | 1.33 s |
+## Usage
 
-For reference, a W4A16 (Marlin) build of the same base model on the same hardware
-with `--enforce-eager` runs at ~14 tok/s single / 86 tok/s @8x / 141 tok/s @16x —
-so NVFP4 on Blackwell is roughly 5x single-stream, 2.3x @8x, and 3x @16x, with
-zero errors at 16-way concurrent 32K contexts.
+### vLLM
 
-## License
+Tested on `vllm/vllm-openai:v0.19.0-cu130` with 2x RTX Pro 6000 Blackwell:
+
+```sh
+vllm serve demon-zombie/MiniMax-M2.7-NVFP4 \
+  --trust-remote-code \
+  --served-model-name MiniMax-M2.7 \
+  --tensor-parallel-size 2 \
+  --max-model-len 196608 \
+  --tool-call-parser minimax_m2 \
+  --reasoning-parser minimax_m2 \
+  --enable-auto-tool-choice \
+  --disable-custom-all-reduce
+```
 
-MIT, inherited from the base model [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7).
+`--disable-custom-all-reduce` is **required** on RTX Pro 6000 (SM120) with TP > 1 — vLLM's custom CUDA all-reduce hangs post-weight-load (workers spin at 100% SM with 0 memory traffic). Not needed on B200. Do **not** pass `--enforce-eager`; CUDA graphs capture cleanly on v0.19.0.
 
-## Credits
+### SGLang
 
-- **Base model:** [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
-- **Quantization toolchain:** [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer)
-- **Inference runtime:** [vLLM](https://github.com/vllm-project/vllm)
+(pending)