Add model card: recipe, hardware support, vLLM launch config, benchmarks

README.md

@@ -0,0 +1,187 @@
+---
+base_model:
+- MiniMaxAI/MiniMax-M2.7
+base_model_relation: quantized
+license: mit
+pipeline_tag: text-generation
+library_name: vllm
+tags:
+- nvfp4
+- fp4
+- quantization
+- modelopt
+- tensorrt-model-optimizer
+- mixture-of-experts
+- moe
+- minimax
+---
+
+# MiniMax-M2.7 — NVFP4 (experts-only) + FP8 KV cache
+
+NVFP4 quantization of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) produced with
+[NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) (`modelopt` 0.43.0rc3).
+
+- **Base model:** MiniMax-M2.7 (230B total / 10B active, 256-expert MoE, 62 layers)
+- **Weight quant:** NVFP4 (W4A4, group size 16) — MoE expert MLPs only (`w1`, `w2`, `w3`)
+- **Attention / router / lm_head:** BF16 (unquantized)
+- **KV cache:** FP8 E4M3 with **static baked scales** (all `k_scale`/`v_scale` = 1.0 after calibration)
+- **Target hardware:** NVIDIA Blackwell (SM100 B200, SM120 RTX Pro 6000) with native NVFP4 tensor cores
+- **On-disk size:** 139.89 GB across 29 safetensors shards (vs. ~460 GB BF16)
+
+## What's quantized
+
+Only the expert MLP projections in each MoE block (`block_sparse_moe.experts.{0..255}.w{1,2,3}`)
+are converted to NVFP4. Everything else — self-attention (`q_proj`/`k_proj`/`v_proj`/`o_proj`),
+router (`block_sparse_moe.gate`), embeddings, layernorms, and `lm_head` — stays in BF16.
+This matches the `excluded_modules` declaration in [`hf_quant_config.json`](./hf_quant_config.json):
+
+```json
+{
+  "quantization": {
+    "quant_algo": "NVFP4",
+    "kv_cache_quant_algo": "FP8",
+    "group_size": 16,
+    "exclude_modules": [
+      "lm_head",
+      "*self_attn*",
+      "*block_sparse_moe.gate*"
+    ]
+  }
+}
+```
+
+## Calibration recipe
+
+- **modelopt recipe:** `nvfp4_experts_only-fp8_kv` (variant of `nvfp4_default-fp8_kv`
+  that leaves attention projections in BF16)
+- **Calibration data:** 1024 samples — 512 from `cnn_dailymail` + 512 from
+  `nvidia/OpenMathReasoning` (math reasoning traces), each truncated to 2048 tokens
+- **Batch size:** 64, `seq_len=2048`
+- **All-expert activation:** `--moe_calib_experts_ratio 1.0` forces every forward pass
+  during calibration to dispatch to all 256 experts in each layer. This is critical —
+  natural top-k routing over 1024 samples leaves roughly 6 experts across the 62 layers
+  completely cold (amax=0), which in turn makes their NVFP4 block scales degenerate and
+  produces characteristic "spurious space" token glitches during inference (e.g.
+  `"mpg_ high"`, `"SPEC. md"`). Forcing full-expert coverage eliminates this class of
+  quality regression entirely.
+- **KV cache scales:** statically calibrated and baked into the safetensors (not computed
+  on-the-fly at serving time). This avoids the silent per-layer `k_scale`/`v_scale = 1.0`
+  fallback in vLLM when KV scales are missing from the checkpoint.
+- **Quantizer fix (modelopt 0.43.0rc3):** 124 `use_constant_amax` quantizers (self-attention
+  Q/K/V/O and router inputs) are missing `_amax` after calibration due to an upstream
+  modelopt bug. Before export we explicitly register `_amax = 448.0` on each one — the
+  correct FP8 E4M3 saturation value. Without this fix the export silently produces a
+  checkpoint that loads but generates garbage.
+- **Hardware:** 4x H100 PCIe (80 GB) — the base BF16 model streams through at ~9 GB/expert
+  layer; the run takes about 90 minutes end-to-end including calibration.
+
+## Quality
+
+Smoke-tested with vLLM v0.19.0 on 2x RTX Pro 6000 (SM120 Blackwell).
+
+- **Prose generation** (5-sentence summary of the Battle of Thermopylae): passes.
+  Correct dates (480 BCE), names (Leonidas, Xerxes, Ephialtes, Thespians, Salamis),
+  coherent narrative, no glitch tokens, `finish_reason=stop`.
+- **Tool calling** (single-turn `get_weather` function): passes. Correct function
+  selected, correct `{"location": "Boston"}` args, `<think>` reasoning properly
+  separated into `reasoning_content` via the `minimax_m2` parser.
+- **No spurious-space tokens** observed across subword-heavy prompts — the cold-expert
+  class of failure is gone.
+
+## Hardware requirements
+
+NVFP4 requires **native FP4 tensor cores**, available on:
+
+| GPU | SM | Status |
+|-----|-----|--------|
+| B200 / GB200 | SM100 | native |
+| B100 | SM100 | native |
+| RTX Pro 6000 (Blackwell) | SM120 | native |
+| DGX Spark (GB10) | SM121 | native (requires 128×128×128 CUTLASS tile) |
+| H100 / H200 | SM90 | **not supported** (no FP4) |
+| A100 | SM80 | **not supported** |
+
+Minimum VRAM for TP=2 at `--max-model-len=196608`: ~90 GB per GPU (FP8 KV cache).
+
+## Serving with vLLM
+
+Tested on **vLLM v0.19.0** with the FLASHINFER attention backend and `FLASHINFER_CUTLASS` NVFP4 MoE kernel (both auto-selected — no env vars needed).
+
+```bash
+python3 -m vllm.entrypoints.openai.api_server \
+  --model demon-zombie/MiniMax-M2.7-NVFP4 \
+  --trust-remote-code \
+  --served-model-name MiniMax-M2.7 \
+  --tensor-parallel-size 2 \
+  --max-model-len 196608 \
+  --max-num-seqs 256 \
+  --gpu-memory-utilization 0.90 \
+  --tool-call-parser minimax_m2 \
+  --reasoning-parser minimax_m2 \
+  --enable-auto-tool-choice \
+  --disable-custom-all-reduce \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+### Required flags
+
+- `--trust-remote-code` — the MiniMax M2 architecture uses custom Python modeling code.
+- `--disable-custom-all-reduce` — **critical on SM120 (RTX Pro 6000) with TP>1**.
+  vLLM's custom CUDA all-reduce hangs post-weight-load on Pro 6000: workers spin at
+  100% SM utilization with 0 memory traffic and minimal power draw, and EngineCore
+  logs `No available shared memory broadcast block found in 60s` every minute. Passing
+  `--disable-custom-all-reduce` falls back to NCCL and avoids the hang. This is required
+  even on tagged v0.19.0-cu130. Not needed on B200.
+- Do **not** pass `--enforce-eager`. Earlier reports suggested eager mode was required
+  on Blackwell — this is no longer true as of vLLM v0.19.0. CUDA graphs capture cleanly
+  (PIECEWISE mixed prefill-decode + FULL decode, 51 + 35 graphs captured) and give
+  roughly 5x single-stream throughput and 2x aggregate throughput at 8x concurrency
+  compared to eager. No accuracy regression.
+
+### Backend auto-selection
+
+vLLM automatically picks:
+
+- **NVFP4 MoE kernel:** `FLASHINFER_CUTLASS` (picked via `enable_sm120_or_later` logic)
+- **KV cache dtype:** `fp8_e4m3` (picked because `kv_cache_quant_algo: FP8` is in
+  `hf_quant_config.json` and the static `k_scale`/`v_scale` tensors are present in the shards)
+- **Attention backend:** `FLASHINFER`
+
+## Performance
+
+Measured on **2x RTX Pro 6000 (SM120 Blackwell, 96 GB each)**, vLLM v0.19.0, TP=2,
+`--disable-custom-all-reduce`, CUDA graphs enabled, `max-model-len=196608`:
+
+### Sequential (single request, gen=500)
+
+| Context | tok/s | TTFT |
+|---------|------:|-----:|
+| warmup  |  80.7 | 1.26 s |
+| 9K      |  77.1 | 1.11 s |
+| 9K cached |  78.5 | 0.40 s |
+| 18K     |  76.0 | 1.25 s |
+| 27K     |  74.3 | 1.43 s |
+| 27K cached |  73.0 | 0.77 s |
+
+### Concurrent (ctx=32K, gen=500, 0 errors)
+
+| Concurrency | agg tok/s | TTFT avg |
+|------------:|----------:|---------:|
+| 8x  | **201.8** | 6.95 s |
+| 16x | **427.8** | 1.33 s |
+
+For reference, a W4A16 (Marlin) build of the same base model on the same hardware
+with `--enforce-eager` runs at ~14 tok/s single / 86 tok/s @8x / 141 tok/s @16x —
+so NVFP4 on Blackwell is roughly 5x single-stream, 2.3x @8x, and 3x @16x, with
+zero errors at 16-way concurrent 32K contexts.
+
+## License
+
+MIT, inherited from the base model [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7).
+
+## Credits
+
+- **Base model:** [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
+- **Quantization toolchain:** [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer)
+- **Inference runtime:** [vLLM](https://github.com/vllm-project/vllm)