DEPLOYMENT.md

# Deployment Guide — MiniMax-M2.7-NVFP4-GB10-AC on 2× DGX Spark

**Purpose**: operator-focused reference for running this model on 2× NVIDIA DGX Spark (GB10) with vLLM. Covers two deployment profiles, measured numbers, and known hardware/framework quirks.

**Target hardware**: 2× DGX Spark (NVIDIA GB10, SM 12.1, 128 GB LPDDR5X unified per node), connected via QSFP56 (ConnectX-7) RoCE.

**Reference runtime**: [`eugr/spark-vllm-docker`](https://github.com/eugr/spark-vllm-docker) nightly (`ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5`), vLLM 0.19.1rc1.dev241, Ray multi-node TP.

**Dates measured**: 2026-04-19, post DGX Spark SoC firmware upgrade to 2.148.24 (gives ~+2 GiB unified memory per node vs 2.144.9).

---

## Two deployment profiles

We tested five tuning phases and converged on two practically useful configs:

| Profile | Config | Best for |
|---|---|---|
| **Agentic** | Marlin NVFP4 MoE + ngram speculative decoding | Tool-use, code generation, chat with repeated tokens (the common case for openclaw-style agent frameworks) |
| **Throughput-stable** | Marlin NVFP4 MoE, no speculative decoding | Batch workloads, long-form novel text, scenarios where predictable per-token latency matters more than wall-clock |

**Default**: [`run_vllm.sh`](./run_vllm.sh) ships with the Agentic profile active. The Throughput-stable profile is documented inline in the script as a commented-out block you can swap in.

---

## Why Marlin MoE on GB10 (both profiles)

GB10's compute capability is SM 12.1. As of April 2026, vLLM's FlashInfer CUTLASS NVFP4 MoE path emits a warning on this GPU:

```
Your GPU does not have native support for FP4 computation but FP4 quantization is being used.
Weight-only FP4 compression will be used leveraging the Marlin kernel.
```

In practice this means the fastest current path on SM 12.1 is the **Marlin** NVFP4 kernel (4-bit weights, FP16 compute, single highly-tuned GEMM per expert). Enabled via these environment variables:

```bash
export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_TEST_FORCE_FP8_MARLIN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
```

The `eugr/spark-vllm-docker` nightly image ships the patched kernels that make this reliable. Community documentation of this unlock: [Avarok-Cybersecurity/dgx-vllm — `NVFP4_BREAKTHROUGH_DGX_SPARK.md`](https://github.com/Avarok-Cybersecurity/dgx-vllm/blob/main/NVFP4_BREAKTHROUGH_DGX_SPARK.md) and [NVIDIA DGX Spark forums](https://forums.developer.nvidia.com/c/dgx-spark).

---

## Measured throughput — `llama-benchy`

Runs: [`llama-benchy`](https://github.com/mostlygeek/llama-bench) v0.3.3, `--runs 3 --no-cache`, single client (concurrency=1), warm model, prefix caching disabled for the benchmark.

### Throughput-stable profile (Marlin, no speculation)

| PP (tok) | TG (tok) | Prefill (tok/s) | Decode (tok/s) | TTFT (ms) |
|---:|---:|---:|---:|---:|
| 512 | 128 | 1,128 | **35.44** | 454 |
| 512 | 256 | 1,248 | **35.86** | 410 |
| 1024 | 128 | 2,049 | 35.03 | 500 |
| 1024 | 256 | 2,132 | 34.50 | 480 |
| 4096 | 128 | 2,817 | 33.76 | 1,454 |
| 4096 | 256 | 3,314 | 33.45 | 1,236 |

API latency (warm): 1.50 ms. Decode peak: **35.86 tok/s** at short prompt length.

### Agentic profile (Marlin + ngram speculative decoding, `num_speculative_tokens=5`)

| PP (tok) | TG (tok) | Prefill (tok/s) | Decode (tok/s) | TTFT (ms) |
|---:|---:|---:|---:|---:|
| 512 | 128 | 1,140 | 27.87 | 449 |
| 512 | 256 | 1,264 | 26.55 | 405 |
| 1024 | 128 | 2,018 | 26.36 | 508 |
| 1024 | 256 | 2,141 | 26.28 | 478 |
| 4096 | 128 | 2,647 | 25.88 | 1,548 |
| 4096 | 256 | 2,888 | 24.88 | 1,419 |

API latency (warm): 1.07 ms. Decode on llama-benchy text: **regresses** vs the Throughput-stable profile.

**Why the regression on this benchmark**: `llama-benchy` generates synthetic prompts with low n-gram repetition. When speculative decoding's acceptance rate is near zero, the cost of running the ngram matcher + parallel verification exceeds the savings. This is the expected "floor" behavior of ngram speculation — it looks like a regression on a synthetic benchmark, and looks very different on real workloads (see next section).

---

## Measured behavior — agentic prompt set (12 hand-crafted prompts)

This is **not a standardized benchmark**. It is 12 prompts we wrote across code-generation, tool-calling, and short chat — designed to approximate the actual traffic an agent framework sends this model. Comparison is: same prompts, same sampling (temp=1.0, top_p=0.95, top_k=40, min_p=0.01 where honored), three profiles:

- **GB10**: the sibling general-chat-calibrated variant, [`saricles/MiniMax-M2.7-NVFP4-GB10`](https://huggingface.co/saricles/MiniMax-M2.7-NVFP4-GB10), Phase 0 config.
- **AC / Throughput-stable**: this repo, Phase 2 config (reported in prior bench; pre-Marlin).
- **AC / Agentic**: this repo, Phase 5 config (current default in `run_vllm.sh`).

### Per-task wall-clock time (lower is better)

| Task | GB10 (s) | AC Throughput-stable (s) | **AC Agentic (s)** |
|---|---:|---:|---:|
| code-01 (HumanEval-style) | 57.3 | 36.3 | **29.0** |
| code-02 (MBPP-style) | 27.6 | 41.0 | **19.3** |
| code-03 (debug) | 41.7 | 40.7 | **32.3** |
| code-04 (async pattern) | 42.7 | 40.3 | **21.2** |
| tool-01 (weather) | 5.7 | 2.3 | **1.8** |
| tool-02 (multi-step booking) | 15.6 | 3.4 | 7.6 ⁎ |
| tool-03 (parallel calls) | 7.1 | 5.0 | **3.3** |
| tool-04 (don't-call-tool trap) | 30.2 | 2.1 | **1.1** |
| chat-01 (reasoning) | 8.2 | 15.5 | **9.9** |
| chat-02 (summarization) | 21.0 | 15.5 | **9.4** |
| chat-03 (creative haiku) | 38.1 | 39.8 | **19.3** |
| chat-04 (instruction following) | 7.9 | 8.9 | 8.8 |
| **Totals** | **303.2** | **250.8** | **162.7** |

**Totals translate to wall-clock time required to complete all 12 tasks serially**, one request at a time. AC Agentic finishes the full set in ~2:43 vs GB10 at 5:03 and AC Throughput-stable at 4:11.

⁎ `tool-02` appears to "regress" for AC Agentic — but the model generated 226 completion tokens there vs 81 on AC Throughput-stable. Per-token decode was still +26% faster (29.8 vs 23.7 tok/s); the model simply chose to say more in response to the same prompt, likely because `min_p=0.01` is silently dropped under speculative decoding (see Caveats).

### Peak decode on the prompt set

- **AC Agentic peak decode: 48.34 tok/s** on `code-04` (async-pattern code generation).
- **AC Agentic average decode: 36.44 tok/s** across all 12 prompts.
- **AC Throughput-stable average decode: 25.20 tok/s** across the same 12 prompts.

These are realized token rates observed by the client over the wire, not kernel-internal numbers.

---

## When to pick which profile

**Pick Agentic (ngram speculation on) when:**
- Traffic is tool-call-heavy or code-generation-heavy
- System prompts and prior turns contain structured patterns (tool names, file paths, JSON keys, variable names) that get repeated in responses
- You care about wall-clock time per agent task, not per-token rate on synthetic benchmarks

**Pick Throughput-stable (ngram off) when:**
- Traffic is long-form novel text (creative writing without repetition, arbitrary Q&A)
- You're publishing tok/s benchmarks against other models on synthetic text
- You want predictable per-token latency irrespective of prompt content
- You need `min_p` sampling to be respected exactly (see Caveats)

**Rule of thumb**: if the model is backing an agent framework, pick Agentic. If it's backing a public chat UI with no tool layer, pick Throughput-stable. You can run both and A/B on your own traffic.

---

## Caveats, observations, known quirks

### 1. GB10 (SM 12.1) does not currently expose native FP4 math to vLLM's Marlin backend

When the Marlin NVFP4 kernel initializes, it emits this warning on DGX Spark:

```
Your GPU does not have native support for FP4 computation but FP4 quantization is being used.
Weight-only FP4 compression will be used leveraging the Marlin kernel.
This may degrade performance for compute-heavy workloads.
```

In practice on GB10 this is the *faster* path — the alternative (FlashInfer CUTLASS MoE FP4) has its own SM 12.1 maturity issues. Upstream work that may change this: [vLLM PR #30906 (NVFP4 W4A16 MoE via Marlin)](https://github.com/vllm-project/vllm/pull/30906) and [vLLM issue #31085 (SM120 native NVFP4 MoE kernels)](https://github.com/vllm-project/vllm/issues/31085).

### 2. `cudagraph_mode: none` is mandatory on multi-node Ray TP

We tested `cudagraph_mode: PIECEWISE` on this exact topology. Capture completed cleanly (the old MiniMax MoE deadlock bug is fixed in this build), but decode **regressed 12-20%** across every prompt/generation size. Root cause: PIECEWISE captures op chunks; on multi-node TP each chunk boundary requires a cross-node sync over QSFP56, and that sync cost exceeds the launch-overhead savings. Conclusion: `cudagraph_mode: none` is the correct setting on dual-Spark Ray. Single-Spark or non-Ray backends may behave differently.

### 3. Under speculative decoding, `min_p` is silently disabled

vLLM logs this explicitly at startup:

```
WARNING: min_p and logit_bias parameters won't work with speculative decoding.
```

MiniMax's recommended sampling includes `min_p=0.01`. When the Agentic profile is active, that parameter has no effect; the sampler falls back to `temperature + top_p + top_k`. Observed consequence: response length distribution shifts slightly (see `tool-02` note above). Output quality in our testing remained within acceptable bounds, but this is worth a side-by-side quality check if your use case is sensitive to `min_p`.

### 4. Async scheduling is disabled on Ray-backed multi-node TP

From vLLM's log: `Async scheduling will be disabled because it is not supported with the 'ray' distributed executor backend.` This is a ~5-10% throughput tax and is structural to cross-node TP on this build. Not user-configurable.

### 5. Expert parallelism (EP) is not recommended on 2-Spark

MiniMax's own documentation recommends `--enable_expert_parallel` for 8-GPU deployments. That advice does not port to 2 nodes. On dual-Spark the forum has observed uneven memory distribution and per-token latency regression when EP is enabled over QSFP56. Stick with `tensor-parallel-size=2`.

### 6. NVFP4 KV cache is not yet supported in vLLM

NVIDIA has documented this capability ([blog post](https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/)), but it's currently TensorRT-LLM only. vLLM tracking issue: [#32220](https://github.com/vllm-project/vllm/issues/32220). Use `--kv-cache-dtype fp8_e4m3` (what the provided `run_vllm.sh` sets).

---

## Reference configuration

### Environment variables (tuned set — both profiles)

```bash
# Model loading
export SAFETENSORS_FAST_GPU=1
export OMP_NUM_THREADS=8
export TORCHINDUCTOR_MAX_AUTOTUNE=0

# Forum-validated Blackwell/SM121 env
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_FLOAT32_MATMUL_PRECISION=high
export VLLM_FLASHINFER_MOE_BACKEND=throughput
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1

# Marlin NVFP4 MoE path (bypasses SM121 FlashInfer CUTLASS MoE maturity issues)
export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_TEST_FORCE_FP8_MARLIN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
```

### vLLM serve CLI (both profiles, differ only in `--speculative-config`)

```bash
vllm serve /models/MiniMax-M2.7-NVFP4-GB10-AC \
  --host 0.0.0.0 --port 30000 \
  --served-model-name minimax-m2.7-ac \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray \
  --gpu-memory-utilization 0.88 \
  --max-model-len 196608 \
  --max-num-seqs 12 \
  --max-num-batched-tokens 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend flashinfer \
  --attention-config.use_trtllm_attention=0 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser minimax_m2 \
  --reasoning-parser minimax_m2_append_think \
  --compilation-config '{"cudagraph_mode":"none","inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false,"max_autotune":false,"max_autotune_gemm":false}}' \
  # Agentic profile only — add this line:
  --speculative-config '{"method":"ngram","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":2}'
```

### Client-side coordination

- `max_tokens` per request: **≥ 16384**. The OpenAI SDK default of 4096 silently truncates tool-call JSON mid-string in MoE models and looks like "the model forgot how to call tools." Bump every client that hits this endpoint.
- Sampling (when not using Agentic speculative path): `temperature=1.0, top_p=0.95, top_k=40, min_p=0.01`.

---

## References and sources

**Forum threads (primary signal for DGX Spark-specific behavior)**:
- [MiniMax M2.7 NVFP4 Recipe & Benchmarks (NVIDIA DGX Spark forum)](https://forums.developer.nvidia.com/t/minimax-m2-7-nfvp4-recipe-benchmarks/366324)
- [Multi-node DGX Spark Cluster benchmarks](https://forums.developer.nvidia.com/t/multi-node-dgx-spark-cluster-4x-k3s-sglang-vllm-connectx-7-sr-iov-full-benchmark-matrix/365555)
- [DGX Spark / GB10 category (landing)](https://forums.developer.nvidia.com/c/dgx-spark)

**Runtime + patches**:
- [Avarok-Cybersecurity/dgx-vllm — NVFP4 breakthrough](https://github.com/Avarok-Cybersecurity/dgx-vllm/blob/main/NVFP4_BREAKTHROUGH_DGX_SPARK.md)
- [eugr/spark-vllm-docker](https://github.com/eugr/spark-vllm-docker)

**vLLM kernel path / issue trackers**:
- [PR #30906 — NVFP4 W4A16 MoE via Marlin](https://github.com/vllm-project/vllm/pull/30906)
- [Issue #31085 — SM120/121 native NVFP4 MoE](https://github.com/vllm-project/vllm/issues/31085)
- [Issue #32220 — NVFP4 KV cache request](https://github.com/vllm-project/vllm/issues/32220)
- [Issue #38971 — NVFP4 MoE backend override](https://github.com/vllm-project/vllm/issues/38971)

**MiniMax official**:
- [MiniMax-M2.7 base model](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
- [MiniMax vLLM deploy guide](https://huggingface.co/MiniMaxAI/MiniMax-M2.7/blob/main/docs/vllm_deploy_guide.md)
- [vLLM Recipes: MiniMax-M2](https://docs.vllm.ai/projects/recipes/en/latest/MiniMax/MiniMax-M2.html)

**NVIDIA technical**:
- [NVFP4 KV cache blog](https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/)
- [Introducing NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)

---

## Changelog

- **2026-04-19**: Initial document. Profiles established from five-phase tuning pass on post-firmware-upgrade GB10.