docs/vllm_deploy_guide.md

# MiniMax-M2.7-MXFP416 vLLM Deployment Guide (RDNA 4 / RX 9700)

## Hardware Requirements

- **GPU:** 8× AMD RX 9700 (RDNA 4 / gfx12xx)
- **Memory:** 128GB+ system RAM for 8-GPU setup
- **OS:** Linux with ROCm 6.x
- **Docker:** RDNA4-compatible vLLM image

## Docker Image

The only validated runtime for this model is [`tcclaviger/vllm22:latest`](https://hub.docker.com/r/tcclaviger/vllm22):

```bash
docker pull tcclaviger/vllm22:latest
```

This image includes:
- Custom Triton attention kernels tuned for RDNA4 (significantly faster than ROCm attention at long context)
- Fixed FP8 KV-cache quantization path (~2× throughput improvement)
- Pre-tuned GEMM configs for RX 9700
- MXFP4-16 kernels compiled for gfx12xx

## System Setup

### GPU Devices

Make sure all GPUs are visible:

```bash
rocm-smi --showid
# Should show: 0, 1, 2, 3, 4, 5, 6, 7
```

### Power Limit (Recommended)

RDNA4 performs best with tuned power limits. Default is ~300W but 210W provides better sustained throughput on multi-GPU setups:

```bash
# Set per-GPU power limit
for i in 0 1 2 3 4 5 6 7; do
    rocm-smi --setpowerlimit $i 210
done
```

> **Note:** At full power (300W) sustained speeds are lower due to thermal throttling. At 210W, sustained generation throughput is consistently higher under multi-user workloads.

## Launching the Server

### Single Container (8 GPUs)

```bash
docker run --name minimax-mxfp416 \
  --rm --tty --ipc=host --shm-size=128g \
  --device /dev/kfd:/dev/kfd \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  --device /dev/dri/renderD130:/dev/dri/renderD130 \
  --device /dev/dri/renderD132:/dev/dri/renderD132 \
  --device /dev/dri/renderD137:/dev/dri/renderD137 \
  --device /dev/dri/renderD138:/dev/dri/renderD138 \
  --device /dev/dri/renderD139:/dev/dri/renderD139 \
  --device /dev/dri/renderD140:/dev/dri/renderD140 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e TRUST_REMOTE_CODE=1 \
  -e PYTORCH_TUNABLEOP_ENABLED=1 \
  -e PYTORCH_TUNABLEOP_TUNING=0 \
  -e PYTORCH_TUNABLEOP_RECORD_UNTUNED=0 \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  -e PYTORCH_HIP_ALLOC_CONF=expandable_segments:True \
  -e GPU_MAX_HW_QUEUES=1 \
  -v /path/to/models:/app/models:ro \
  -p 8000:8000 \
  tcclaviger/vllm22:latest \
  bash -c "cp /app/models/vllm22_minimax_m2.py /app/vllm/vllm/model_executor/models/minimax_m2.py && \
    pip install -q sentencepiece && \
    exec vllm serve \
      /app/models/MiniMax-M2.7-MXFP416 \
      --served-model-name minimax-m2.7-mxfp416 \
      --host 0.0.0.0 --port 8000 \
      --trust-remote-code \
      --tensor-parallel-size 8 \
      --enable-expert-parallel \
      --disable-cascade-attn \
      --reasoning-parser minimax_m2 \
      --enable-auto-tool-choice \
      --tool-call-parser minimax_m2 \
      --enable-prefix-caching \
      --gpu-memory-utilization 0.93 \
      --max-model-len 180000 \
      --max-num-seqs 48 \
      --max-num-batched-tokens 2048 \
      --kv-cache-dtype fp8_e4m3 \
      --attention-backend TRITON_ATTN \
      --override-generation-config '{\"max_tokens\": 16384}'"
```

### Key vLLM Flags Explained

| Flag | Value | Purpose |
|------|-------|---------|
| `--tensor-parallel-size` | 8 | Split model across 8 GPUs |
| `--enable-expert-parallel` | | Enable expert-parallel distribution |
| `--disable-cascade-attn` | | Disable cascade attention for MoE layers |
| `--attention-backend` | TRITON_ATTN | Use Triton kernels (10× faster than ROCm on RDNA4) |
| `--kv-cache-dtype` | fp8_e4m3 | FP8 KV cache (~50% memory savings) |
| `--enable-prefix-caching` | | Cache common prefixes (93%+ hit rate observed) |
| `--max-model-len` | 180000 | 180k context |
| `--max-num-seqs` | 48 | Max concurrent sequences |
| `--max-num-batched-tokens` | 2048 | Max tokens per batch |
| `--gpu-memory-utilization` | 0.93 | Use 93% of GPU memory |

> **Important:** The `--disable-cascade-attn` flag is required for MoE models. Without it, the model will produce incorrect outputs.

### Running with Patches

If you have custom model patches:

```bash
-v /path/to/patches:/patches:ro \
```

The Docker entry point copies `vllm22_minimax_m2.py` to the vLLM model directory before launching. This adds MXFP4-16 support for MiniMax-M2.7.

## Performance Notes

### Observed Performance (8× RX 9700, 210W power limit)

- **Generation throughput:** 30–35 tokens/s
- **Prefill throughput:** 2000+ tokens/s (with prefix caching)
- **Prefix cache hit rate:** ~93%
- **KV cache usage:** 25–33% typical at 180k context
- **Max concurrent users:** 4–5 at full 180k context

### KV Cache Capacity

With 8× RX 9700 and FP8 KV cache:
- **KV cache memory:** 11.35 GiB
- **KV cache tokens:** ~768K tokens
- **Max context per request:** 180,000 tokens
- **Max concurrent at 180k:** ~4 requests

### Model Loading

- **Weight loading time:** ~42 seconds
- **Memory per GPU (TP8):** ~17.5 GiB
- **Torch compile warmup:** ~37 seconds

## Testing the Deployment

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

completion = client.chat.completions.create(
    model="minimax-m2.7-mxfp416",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what MXFP4 quantization is in one sentence."}
    ],
    temperature=1.0,
    max_tokens=256,
)

print(completion.choices[0].message.content)
```

## Troubleshooting

### "expandable_segments not supported"

This warning is benign on ROCm. The model runs correctly despite the warning.

### Low throughput at long context

Ensure `TRITON_ATTN` backend is active. Default ROCm attention is 10× slower on RDNA4 at long context.

### Thermal throttling

If sustained throughput degrades over time, reduce power limit to 210W per GPU:
```bash
rocm-smi --setpowerlimit 0 210
```

### Model fails to load

Ensure `--trust-remote-code` is set and the model path is correct. The custom model file (`vllm22_minimax_m2.py`) must be copied before vLLM loads the model.