---
license: mit
base_model: mitomtuna/MiMo-V2.5-0703-NVFP4
tags:
- mimo
- nvfp4
- mxfp8
- modelopt
- tensor-parallel
- tp3
- multimodal
- vllm
- dflash
---

# MiMo-V2.5-0703-NVFP4-TP3

**[mitomtuna/MiMo-V2.5-0703-NVFP4](https://huggingface.co/mitomtuna/MiMo-V2.5-0703-NVFP4)
transformed to run at tensor-parallel size 3** (e.g. 3× 96 GB GPUs) — a
geometry the source cannot serve (64 attention heads, 4/8 KV heads, MoE
intermediate 2048: none divisible by 3).

This is the **updated 2026-07-03 MiMo-V2.5 target** (the exact weights the
DFlash drafter was trained against) — the successor of
[mitomtuna/MiMo-V2.5-NVFP4-TP3](https://huggingface.co/mitomtuna/MiMo-V2.5-NVFP4-TP3),
which was built from the original April base.

The transform is **mathematically exact** — no requantization, no fine-tuning.
Full-attn 64Q/4KV → 72Q/9KV (bit-identical KV duplication, slot map
`[0,0,0,1,1,2,2,3,3]`), SWA 64Q/8KV → 72Q/9KV (`[0,0,1,2,3,4,5,6,7]`), padded
Q heads have zeroed q rows AND zeroed o_proj columns (exact-zero contribution),
MoE expert intermediate 2048 → 2304 and dense MLP 16384 → 16512 via
group-aligned zero padding of the packed NVFP4/MXFP8 tensors. All original
quantization groups are byte-identical. `transform_checkpoint.py` (included)
is the exact script that produced this repo.

Use with the matching DFlash drafter (same transform, drafter weights are
Xiaomi's originals): [mitomtuna/MiMo-V2.5-DFlash-TP3](https://huggingface.co/mitomtuna/MiMo-V2.5-DFlash-TP3).

## Running with vLLM

```
docker pull ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash
```

```bash
docker run --gpus '"device=0,1,2"' --ipc host --network host --init --shm-size 32g \
  -v $HF_CACHE:/root/.cache/huggingface \
  -v mimo-jit-cache:/cache \
  -e VLLM_USE_V2_MODEL_RUNNER=0 \
  -e VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS=1 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SAFETENSORS_FAST_GPU=1 \
  -e CUDA_DEVICE_MAX_CONNECTIONS=32 \
  -e OMP_NUM_THREADS=16 \
  -e CUTE_DSL_ARCH=sm_120a \
  -e NCCL_P2P_DISABLE=0 \
  -e NCCL_SHM_DISABLE=0 \
  -e NCCL_P2P_LEVEL=SYS \
  -e NCCL_MIN_NCHANNELS=8 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e NCCL_CUMEM_HOST_ENABLE=0 \
  ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash \
  vllm serve mitomtuna/MiMo-V2.5-0703-NVFP4-TP3 \
    --served-model-name mimo-v2.5 \
    --trust-remote-code \
    --tensor-parallel-size 3 \
    --kv-cache-dtype fp8 \
    --block-size 64 \
    --gpu-memory-utilization 0.965 \
    --max-model-len 1048576 \
    --max-num-seqs 32 \
    --max-num-batched-tokens 4096 \
    --max-cudagraph-capture-size 32 \
    --attention-backend TRITON_ATTN \
    --kernel-config.moe_backend flashinfer_cutlass \
    --kernel-config.linear_backend b12x \
    --mm-encoder-tp-mode data \
    --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice \
    --compilation-config '{"cudagraph_mode":"PIECEWISE","custom_ops":["all"]}' \
    --async-scheduling --no-scheduler-reserve-full-isl \
    --enable-chunked-prefill --enable-prefix-caching \
    --disable-custom-all-reduce \
    --speculative-config '{"model":"mitomtuna/MiMo-V2.5-DFlash-TP3","method":"dflash","num_speculative_tokens":7,"num_speculative_tokens_per_batch_size":[[1,4,7],[5,32,3]]}'
```

This is the exact production configuration the numbers below were measured on
(only host/port and model paths differ).

Notes:

- **Read "Host setup" below first** — without driver-enabled GPU P2P the
  decode numbers here are not reproducible.
- fp8 KV (the command above) needs a **2026-07-13+ image pull** — older builds
  silently ignore `--kv-cache-dtype` and the 1M pool won't fit. It gives
  **1,358,484 tokens** (1.30× at 1M, 2.0× bf16), acceptance at parity, decode
  7–12% faster at 128k+ — but cold long-context prefill 30–50% slower (the
  b12x prefill fast path is 16-bit-only). bf16 alternative:
  `--kv-cache-dtype auto --max-model-len 524288` → 668,489 tokens, fastest
  prefill.
- 0.975 util OOMs a ~4.8 GiB init transient — 0.965 assumes dedicated,
  otherwise-idle 96 GB GPUs; back off to 0.93–0.95 if anything else shares them.
- `--mm-encoder-tp-mode data` is **required** at TP3 (vision tower has 32
  heads, not divisible by 3; encoders run replicated). Text/image/audio/video
  all work.
- Decode throughput ≈ 0.75–0.86× of the same engine at TP4; KV duplication
  costs ~3× per-token KV per rank vs TP4 (fp8 KV halves that back).

## Host setup (required for full multi-GPU speed)

Tensor-parallel decode is allreduce-latency-bound. On RTX PRO 6000 Blackwell
the driver disables PCIe peer access by default, so NCCL silently falls back
to shared-memory copies and decode is dramatically slower. To get the fast
lane these numbers were measured on:

1. Force P2P in the driver — `/etc/modprobe.d/nvidia-p2p-override.conf`:
   ```
   options nvidia NVreg_RegistryDwords="ForceP2P=0x11;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1"
   ```
   then `update-initramfs -u` and reboot. Do **not** add the older
   `RMForceP2PType=1;RMPcieP2PType=2` dwords seen in some recipes — they break
   cross-process cuMem IPC on NCCL ≥ 2.29.
2. BIOS: Resizable BAR and Above-4G decoding ON; IOMMU OFF (or ACS override) —
   an active IOMMU degrades or breaks PCIe P2P.
3. Container env (in the command above): `NCCL_P2P_LEVEL=SYS` so NCCL uses P2P
   across PCIe root complexes, and `NCCL_CUMEM_ENABLE=0` (legacy cudaIpc
   transport — most reliable with forced P2P).
4. Keep `--disable-custom-all-reduce`: plain NCCL over P2P is the fast path on
   this platform (vLLM's custom-allreduce kernels crash or regress here).

Verify: `nvidia-smi topo -p2p r` should show all GPU pairs OK, and a boot with
`NCCL_DEBUG=INFO` should log channels `via P2P`, not `via SHM`.

## Credits

- Original model + DFlash drafter + updated target: [XiaomiMiMo](https://huggingface.co/XiaomiMiMo)
- Quantization recipe/seed/pipeline: [lukealonso](https://huggingface.co/lukealonso) /
  [local-inference-lab](https://github.com/local-inference-lab/quant-toolkit)
- TP4→TP3 transform: `transform_checkpoint.py` in this repo

## Validation (this exact checkpoint, RTX PRO 6000 Blackwell)

# 0703-target requant validation — 2026-07-08T07:40:08+00:00

## Acceptance (reason_accept_check.py, same prompt/protocol)
```
--- baseline: prod TP3, OLD target (captured in prep):
reasoning ctx0: 412.3 t/s, accept_len 5.91/8 (2500 tokens, 423 drafts)
--- mimo-vllm-val (NEW target):
reasoning ctx0: 488.0 t/s, accept_len 5.69/8 (2500 tokens, 439 drafts)
--- mimo-tp3-val (NEW target):
reasoning ctx0: 381.5 t/s, accept_len 5.71/8 (2500 tokens, 438 drafts)
```

## Needle (90k)
```
--- mimo-vllm-val:
prompt_tokens=89935 completion=99
answer: 'PERSIMMON-42'
NEEDLE PASS
--- mimo-tp3-val:
prompt_tokens=89935 completion=103
answer: 'PERSIMMON-42'
NEEDLE PASS
```

## KV cache pool measured at boot (this checkpoint, this command)
```
--- TP4 (util 0.85, max-model-len 1048576):
(Worker_TP0 pid=247) INFO 07-08 07:35:33 [gpu_worker.py:517] Available KV cache memory: 29.77 GiB
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2202] GPU KV cache size: 3,877,291 tokens
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2203] Maximum concurrency for 1,048,576 tokens per request: 3.70x
--- TP3, fp8 KV (util 0.965, max-model-len 1048576 — current command above):
(Worker_TP0 pid=186) INFO 07-13 06:21:12 [gpu_worker.py:517] Available KV cache memory: 14.87 GiB
(EngineCore pid=118) INFO 07-13 06:21:12 [kv_cache_utils.py:2202] GPU KV cache size: 1,358,484 tokens
(EngineCore pid=118) INFO 07-13 06:21:12 [kv_cache_utils.py:2203] Maximum concurrency for 1,048,576 tokens per request: 1.30x
--- TP3, bf16 KV (util 0.965, max-model-len 524288 — the "auto" alternative):
(Worker_TP0 pid=241) INFO 07-08 07:39:30 [gpu_worker.py:517] Available KV cache memory: 14.9 GiB
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2202] GPU KV cache size: 668,489 tokens
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2203] Maximum concurrency for 524,288 tokens per request: 1.28x
```

## Smoke responses: see s5-smoke-*.json
Result: ALL PASS