Deployment Guide — MiniMax-M2.7-NVFP4-GB10-AC on 2× DGX Spark
Purpose: operator-focused reference for running this model on 2× NVIDIA DGX Spark (GB10) with vLLM. Covers two deployment profiles, measured numbers, and known hardware/framework quirks.
Target hardware: 2× DGX Spark (NVIDIA GB10, SM 12.1, 128 GB LPDDR5X unified per node), connected via QSFP56 (ConnectX-7) RoCE.
Reference runtime: eugr/spark-vllm-docker nightly (ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5), vLLM 0.19.1rc1.dev241, Ray multi-node TP.
Dates measured: 2026-04-19, post DGX Spark SoC firmware upgrade to 2.148.24 (gives ~+2 GiB unified memory per node vs 2.144.9).
Two deployment profiles
We tested five tuning phases and converged on two practically useful configs:
| Profile | Config | Best for |
|---|---|---|
| Agentic | Marlin NVFP4 MoE + ngram speculative decoding | Tool-use, code generation, chat with repeated tokens (the common case for openclaw-style agent frameworks) |
| Throughput-stable | Marlin NVFP4 MoE, no speculative decoding | Batch workloads, long-form novel text, scenarios where predictable per-token latency matters more than wall-clock |
Default: run_vllm.sh ships with the Agentic profile active. The Throughput-stable profile is documented inline in the script as a commented-out block you can swap in.
Why Marlin MoE on GB10 (both profiles)
GB10's compute capability is SM 12.1. As of April 2026, vLLM's FlashInfer CUTLASS NVFP4 MoE path emits a warning on this GPU:
Your GPU does not have native support for FP4 computation but FP4 quantization is being used.
Weight-only FP4 compression will be used leveraging the Marlin kernel.
In practice this means the fastest current path on SM 12.1 is the Marlin NVFP4 kernel (4-bit weights, FP16 compute, single highly-tuned GEMM per expert). Enabled via these environment variables:
export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_TEST_FORCE_FP8_MARLIN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
The eugr/spark-vllm-docker nightly image ships the patched kernels that make this reliable. Community documentation of this unlock: Avarok-Cybersecurity/dgx-vllm — NVFP4_BREAKTHROUGH_DGX_SPARK.md and NVIDIA DGX Spark forums.
Measured throughput — llama-benchy
Runs: llama-benchy v0.3.3, --runs 3 --no-cache, single client (concurrency=1), warm model, prefix caching disabled for the benchmark.
Throughput-stable profile (Marlin, no speculation)
| PP (tok) | TG (tok) | Prefill (tok/s) | Decode (tok/s) | TTFT (ms) |
|---|---|---|---|---|
| 512 | 128 | 1,128 | 35.44 | 454 |
| 512 | 256 | 1,248 | 35.86 | 410 |
| 1024 | 128 | 2,049 | 35.03 | 500 |
| 1024 | 256 | 2,132 | 34.50 | 480 |
| 4096 | 128 | 2,817 | 33.76 | 1,454 |
| 4096 | 256 | 3,314 | 33.45 | 1,236 |
API latency (warm): 1.50 ms. Decode peak: 35.86 tok/s at short prompt length.
Agentic profile (Marlin + ngram speculative decoding, num_speculative_tokens=5)
| PP (tok) | TG (tok) | Prefill (tok/s) | Decode (tok/s) | TTFT (ms) |
|---|---|---|---|---|
| 512 | 128 | 1,140 | 27.87 | 449 |
| 512 | 256 | 1,264 | 26.55 | 405 |
| 1024 | 128 | 2,018 | 26.36 | 508 |
| 1024 | 256 | 2,141 | 26.28 | 478 |
| 4096 | 128 | 2,647 | 25.88 | 1,548 |
| 4096 | 256 | 2,888 | 24.88 | 1,419 |
API latency (warm): 1.07 ms. Decode on llama-benchy text: regresses vs the Throughput-stable profile.
Why the regression on this benchmark: llama-benchy generates synthetic prompts with low n-gram repetition. When speculative decoding's acceptance rate is near zero, the cost of running the ngram matcher + parallel verification exceeds the savings. This is the expected "floor" behavior of ngram speculation — it looks like a regression on a synthetic benchmark, and looks very different on real workloads (see next section).
Measured behavior — agentic prompt set (12 hand-crafted prompts)
This is not a standardized benchmark. It is 12 prompts we wrote across code-generation, tool-calling, and short chat — designed to approximate the actual traffic an agent framework sends this model. Comparison is: same prompts, same sampling (temp=1.0, top_p=0.95, top_k=40, min_p=0.01 where honored), three profiles:
- GB10: the sibling general-chat-calibrated variant,
saricles/MiniMax-M2.7-NVFP4-GB10, Phase 0 config. - AC / Throughput-stable: this repo, Phase 2 config (reported in prior bench; pre-Marlin).
- AC / Agentic: this repo, Phase 5 config (current default in
run_vllm.sh).
Per-task wall-clock time (lower is better)
| Task | GB10 (s) | AC Throughput-stable (s) | AC Agentic (s) |
|---|---|---|---|
| code-01 (HumanEval-style) | 57.3 | 36.3 | 29.0 |
| code-02 (MBPP-style) | 27.6 | 41.0 | 19.3 |
| code-03 (debug) | 41.7 | 40.7 | 32.3 |
| code-04 (async pattern) | 42.7 | 40.3 | 21.2 |
| tool-01 (weather) | 5.7 | 2.3 | 1.8 |
| tool-02 (multi-step booking) | 15.6 | 3.4 | 7.6 ⁎ |
| tool-03 (parallel calls) | 7.1 | 5.0 | 3.3 |
| tool-04 (don't-call-tool trap) | 30.2 | 2.1 | 1.1 |
| chat-01 (reasoning) | 8.2 | 15.5 | 9.9 |
| chat-02 (summarization) | 21.0 | 15.5 | 9.4 |
| chat-03 (creative haiku) | 38.1 | 39.8 | 19.3 |
| chat-04 (instruction following) | 7.9 | 8.9 | 8.8 |
| Totals | 303.2 | 250.8 | 162.7 |
Totals translate to wall-clock time required to complete all 12 tasks serially, one request at a time. AC Agentic finishes the full set in ~2:43 vs GB10 at 5:03 and AC Throughput-stable at 4:11.
⁎ tool-02 appears to "regress" for AC Agentic — but the model generated 226 completion tokens there vs 81 on AC Throughput-stable. Per-token decode was still +26% faster (29.8 vs 23.7 tok/s); the model simply chose to say more in response to the same prompt, likely because min_p=0.01 is silently dropped under speculative decoding (see Caveats).
Peak decode on the prompt set
- AC Agentic peak decode: 48.34 tok/s on
code-04(async-pattern code generation). - AC Agentic average decode: 36.44 tok/s across all 12 prompts.
- AC Throughput-stable average decode: 25.20 tok/s across the same 12 prompts.
These are realized token rates observed by the client over the wire, not kernel-internal numbers.
When to pick which profile
Pick Agentic (ngram speculation on) when:
- Traffic is tool-call-heavy or code-generation-heavy
- System prompts and prior turns contain structured patterns (tool names, file paths, JSON keys, variable names) that get repeated in responses
- You care about wall-clock time per agent task, not per-token rate on synthetic benchmarks
Pick Throughput-stable (ngram off) when:
- Traffic is long-form novel text (creative writing without repetition, arbitrary Q&A)
- You're publishing tok/s benchmarks against other models on synthetic text
- You want predictable per-token latency irrespective of prompt content
- You need
min_psampling to be respected exactly (see Caveats)
Rule of thumb: if the model is backing an agent framework, pick Agentic. If it's backing a public chat UI with no tool layer, pick Throughput-stable. You can run both and A/B on your own traffic.
Caveats, observations, known quirks
1. GB10 (SM 12.1) does not currently expose native FP4 math to vLLM's Marlin backend
When the Marlin NVFP4 kernel initializes, it emits this warning on DGX Spark:
Your GPU does not have native support for FP4 computation but FP4 quantization is being used.
Weight-only FP4 compression will be used leveraging the Marlin kernel.
This may degrade performance for compute-heavy workloads.
In practice on GB10 this is the faster path — the alternative (FlashInfer CUTLASS MoE FP4) has its own SM 12.1 maturity issues. Upstream work that may change this: vLLM PR #30906 (NVFP4 W4A16 MoE via Marlin) and vLLM issue #31085 (SM120 native NVFP4 MoE kernels).
2. cudagraph_mode: none is mandatory on multi-node Ray TP
We tested cudagraph_mode: PIECEWISE on this exact topology. Capture completed cleanly (the old MiniMax MoE deadlock bug is fixed in this build), but decode regressed 12-20% across every prompt/generation size. Root cause: PIECEWISE captures op chunks; on multi-node TP each chunk boundary requires a cross-node sync over QSFP56, and that sync cost exceeds the launch-overhead savings. Conclusion: cudagraph_mode: none is the correct setting on dual-Spark Ray. Single-Spark or non-Ray backends may behave differently.
3. Under speculative decoding, min_p is silently disabled
vLLM logs this explicitly at startup:
WARNING: min_p and logit_bias parameters won't work with speculative decoding.
MiniMax's recommended sampling includes min_p=0.01. When the Agentic profile is active, that parameter has no effect; the sampler falls back to temperature + top_p + top_k. Observed consequence: response length distribution shifts slightly (see tool-02 note above). Output quality in our testing remained within acceptable bounds, but this is worth a side-by-side quality check if your use case is sensitive to min_p.
4. Async scheduling is disabled on Ray-backed multi-node TP
From vLLM's log: Async scheduling will be disabled because it is not supported with the 'ray' distributed executor backend. This is a ~5-10% throughput tax and is structural to cross-node TP on this build. Not user-configurable.
5. Expert parallelism (EP) is not recommended on 2-Spark
MiniMax's own documentation recommends --enable_expert_parallel for 8-GPU deployments. That advice does not port to 2 nodes. On dual-Spark the forum has observed uneven memory distribution and per-token latency regression when EP is enabled over QSFP56. Stick with tensor-parallel-size=2.
6. NVFP4 KV cache is not yet supported in vLLM
NVIDIA has documented this capability (blog post), but it's currently TensorRT-LLM only. vLLM tracking issue: #32220. Use --kv-cache-dtype fp8_e4m3 (what the provided run_vllm.sh sets).
Reference configuration
Environment variables (tuned set — both profiles)
# Model loading
export SAFETENSORS_FAST_GPU=1
export OMP_NUM_THREADS=8
export TORCHINDUCTOR_MAX_AUTOTUNE=0
# Forum-validated Blackwell/SM121 env
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_FLOAT32_MATMUL_PRECISION=high
export VLLM_FLASHINFER_MOE_BACKEND=throughput
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
# Marlin NVFP4 MoE path (bypasses SM121 FlashInfer CUTLASS MoE maturity issues)
export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_TEST_FORCE_FP8_MARLIN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
vLLM serve CLI (both profiles, differ only in --speculative-config)
vllm serve /models/MiniMax-M2.7-NVFP4-GB10-AC \
--host 0.0.0.0 --port 30000 \
--served-model-name minimax-m2.7-ac \
--tensor-parallel-size 2 \
--distributed-executor-backend ray \
--gpu-memory-utilization 0.88 \
--max-model-len 196608 \
--max-num-seqs 12 \
--max-num-batched-tokens 32768 \
--kv-cache-dtype fp8_e4m3 \
--attention-backend flashinfer \
--attention-config.use_trtllm_attention=0 \
--enable-prefix-caching \
--enable-chunked-prefill \
--trust-remote-code \
--enable-auto-tool-choice --tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2_append_think \
--compilation-config '{"cudagraph_mode":"none","inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false,"max_autotune":false,"max_autotune_gemm":false}}' \
# Agentic profile only — add this line:
--speculative-config '{"method":"ngram","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":2}'
Client-side coordination
max_tokensper request: ≥ 16384. The OpenAI SDK default of 4096 silently truncates tool-call JSON mid-string in MoE models and looks like "the model forgot how to call tools." Bump every client that hits this endpoint.- Sampling (when not using Agentic speculative path):
temperature=1.0, top_p=0.95, top_k=40, min_p=0.01.
References and sources
Forum threads (primary signal for DGX Spark-specific behavior):
- MiniMax M2.7 NVFP4 Recipe & Benchmarks (NVIDIA DGX Spark forum)
- Multi-node DGX Spark Cluster benchmarks
- DGX Spark / GB10 category (landing)
Runtime + patches:
vLLM kernel path / issue trackers:
- PR #30906 — NVFP4 W4A16 MoE via Marlin
- Issue #31085 — SM120/121 native NVFP4 MoE
- Issue #32220 — NVFP4 KV cache request
- Issue #38971 — NVFP4 MoE backend override
MiniMax official:
NVIDIA technical:
Changelog
- 2026-04-19: Initial document. Profiles established from five-phase tuning pass on post-firmware-upgrade GB10.