README: update Running section with tuned Phase 2 config + env vars, add measured bench numbers + qualitative tool-use comparison

README.md

@@ -96,30 +96,56 @@ The 7th dataset (`ultrachat_200k`) is intentional: without a general-chat anchor
 
 At 141.05 GB this model does not fit in a single DGX Spark's 128 GB unified memory. It is intended to run with `tensor-parallel-size=2` across two Sparks connected via their ConnectX-7 200 GbE link, orchestrated by Ray. The community reference container is [`eugr/spark-vllm-docker`](https://github.com/eugr/spark-vllm-docker).
 
-Deployment, environment variables, Ray cluster setup, and the vLLM server invocation are **identical to `NVFP4-GB10`** — only the per-layer NVFP4 scales differ between the two. See the full [`saricles/MiniMax-M2.7-NVFP4-GB10` README](https://huggingface.co/saricles/MiniMax-M2.7-NVFP4-GB10) for the RoCE interface envvars, Ray head/worker commands, and the full `vllm serve` flag set.
+A ready-to-run wrapper is included: [`run_vllm.sh`](./run_vllm.sh) — exports the forum-validated environment variables and invokes `vllm serve` with the tuned flag set documented below.
 
-Short version (matching `NVFP4-GB10` exactly, only the `/models/...` path differs):
+### Tuned environment variables (measured 2026-04-19 on 2× GB10)
+
+```bash
+export VLLM_USE_FLASHINFER_MOE_FP4=1              # Blackwell-native MoE FP4 path (biggest single win)
+export VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass
+export VLLM_FLASHINFER_MOE_BACKEND=throughput
+export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1            # unlocks full 196K advertised
+export VLLM_FLOAT32_MATMUL_PRECISION=high         # TF32 tensor-core accel for residual fp32 ops
+export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
+export SAFETENSORS_FAST_GPU=1
+export OMP_NUM_THREADS=8
+```
+
+### vLLM serve invocation
 
 ```bash
 vllm serve /models/MiniMax-M2.7-NVFP4-GB10-AC \
   --host 0.0.0.0 --port 30000 \
-  --served-model-name minimax-m2.7 \
+  --served-model-name minimax-m2.7-ac \
   --tensor-parallel-size 2 \
   --distributed-executor-backend ray \
-  --gpu-memory-utilization 0.85 \
+  --gpu-memory-utilization 0.88 \
   --max-model-len 196608 \
   --kv-cache-dtype fp8_e4m3 \
   --attention-backend flashinfer \
+  --attention-config.use_trtllm_attention=0 \
   --enable-prefix-caching \
   --enable-chunked-prefill \
-  --max-num-seqs 64 \
-  --max-num-batched-tokens 8192 \
+  --max-num-seqs 12 \
+  --max-num-batched-tokens 32768 \
   --trust-remote-code \
   --enable-auto-tool-choice --tool-call-parser minimax_m2 \
   --reasoning-parser minimax_m2_append_think \
   --compilation-config '{"cudagraph_mode":"none","inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false,"max_autotune":false,"max_autotune_gemm":false}}'
 ```
 
+### Why these flags (the non-obvious ones)
+
+- **`VLLM_USE_FLASHINFER_MOE_FP4=1`** — many older images/configs defaulted `0`, which forces the slower CUTLASS MoE path. Flipping to `1` engages the Blackwell-native FlashInfer MoE FP4 kernel. On our dual-Spark setup this alone delivered the bulk of the measurable decode/prefill gains.
+- **`cudagraph_mode=none`** — **intentional.** We tested `PIECEWISE` in this build (the historical MiniMax MoE deadlock is fixed) and measured a 12–20% decode *regression* on dual-node Ray TP. Root cause: PIECEWISE captures op chunks, and across multi-node TP each chunk boundary forces a cross-node sync whose cost exceeds the launch-overhead savings. `none` remains the right choice on this topology. Single-Spark or non-Ray backends may behave differently — retest if you change the distributed executor.
+- **`--attention-config.use_trtllm_attention=0`** — community-flagged unstable for MiniMax MoE. Stick with the FlashInfer path.
+- **`--gpu-memory-utilization 0.88`** — 0.85 was the conservative default; 0.88 reclaims the headroom freed by the SoC firmware update (`fwupdmgr` SoC ≥ 2.148.24 on GB10 gives ~+2 GiB per Spark).
+- **`--max-num-seqs 12`, `--max-num-batched-tokens 32768`** — balanced for dual-Spark TP=2 with MoE. The 32K batched-tokens ceiling sits at the prefill-compute saturation point for GB10 before the QSFP56 all-reduce becomes the bottleneck.
+
+### Client-side tips
+
+Every client that calls this endpoint should set `max_tokens ≥ 16384`. The OpenAI SDK's default of 4096 will silently truncate tool-call JSON mid-string, which appears as "model forgot how to use tools" but is actually just a clipped response. Bump it.
+
 ## When to choose `-AC` vs `NVFP4-GB10`
 
 - **Use `-AC`** for: agent frameworks (OpenClaw, Aider, Claude Code-style), tool-calling workloads, code-generation assistants, multi-turn reasoning over code/math.
@@ -129,9 +155,39 @@ Both variants are mechanically compatible (same vLLM invocation, same `compresse
 
 ## Performance
 
-Benchmark results: *TBD — to be filled in after benchmarking on 2× DGX Spark.*
+Benchmarked on **2× NVIDIA DGX Spark (GB10)**, TP=2 via Ray over QSFP56 RoCE, using [`llama-benchy`](https://github.com/mostlygeek/llama-bench) v0.3.3. Measured 2026-04-19 with the tuned config shown above (including `VLLM_USE_FLASHINFER_MOE_FP4=1`, SoC firmware ≥2.148.24, `--gpu-memory-utilization 0.88`).
+
+### Throughput
+
+| Prompt (tok) | Gen (tok) | Prefill (tok/s) | Decode (tok/s) | TTFT (ms) |
+|---:|---:|---:|---:|---:|
+| 512  | 128 | 1,179 | **28.82** | 434 |
+| 512  | 256 | 1,299 | **28.71** | 394 |
+| 1024 | 128 | 2,061 | 27.83 | 497 |
+| 1024 | 256 | 2,322 | 26.02 | 441 |
+| 4096 | 128 | 3,487 | 25.72 | 1,175 |
+| 4096 | 256 | **4,025** | 25.76 | 1,018 |
+
+API latency: 1.56 ms. 3 runs per configuration, warm model, prefix caching disabled for the benchmark. Decode is memory-bandwidth-limited at longer contexts on GB10 — the 28.8 tok/s peak decode is the short-context sweet spot where FlashInfer MoE FP4 kernels dominate.
+
+### Qualitative — agentic / tool-use behavior
+
+Hand-crafted prompt set (not a standard benchmark; purposeful smoke tests for the calibration-dataset fingerprint). Same prompts, same sampling (temp=1.0, top_p=0.95, top_k=40, min_p=0.01), compared against the general-chat-calibrated [`NVFP4-GB10`](https://huggingface.co/saricles/MiniMax-M2.7-NVFP4-GB10) sibling variant:
+
+| Task | `NVFP4-GB10` tokens | `NVFP4-GB10-AC` tokens | Wall-clock speedup (AC) |
+|---|---:|---:|---:|
+| "Answer directly; don't call the provided tool" (trap) | 718 | **44** | **14.7×** |
+| Multi-step meeting booking (3 tools) | 385 | **81** | 4.6× |
+| Weather (single tool) | 73 | **51** | 2.5× |
+| Parallel stock prices (parallel tool calls) | 176 | **121** | 1.4× |
+
+**AC is measurably more decisive on tool-use tasks** — it emits cleaner, shorter tool calls and, crucially, doesn't over-invoke tools when direct answers suffice. Same base model, same quantization scheme, different calibration mix — the agentic/coder calibration shows up as measurable behavioral tightening on tool-use distributions. On plain chat, wall-clock differences are within typical generation variance.
+
+### Notes
 
-Expected: decode throughput within ±1 tok/s of `NVFP4-GB10` (quantization format identical; only activation scales differ), with measurable quality improvements on agentic/coder evals.
+- Raw throughput (decode/prefill) is within noise of `NVFP4-GB10` as expected — quant format is identical, only activation scales differ.
+- The meaningful delta between AC and GB10 is **qualitative on agentic tasks**, not numeric on raw throughput.
+- For published standardized benchmarks (HumanEval, BFCL, MT-Bench, WildClawBench), see forthcoming evaluation runs.
 
 ## Recommended Sampling Parameters