Upload README.md

README.md

@@ -0,0 +1,279 @@
+---
+pipeline_tag: text-generation
+license: other
+license_name: other
+license_link: https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE
+library_name: transformers
+base_model: MiniMaxAI/MiniMax-M2.7
+tags:
+- minimax_m2
+- mxfp4_16
+- text-generation
+- moe
+- mixture-of-experts
+- custom_code
+- 8-bit precision
+---
+
+## `mxfp4_16` Quantization of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
+
+**Runtime:** Requires [`tcclaviger/vllm22:latest`](https://hub.docker.com/r/tcclaviger/vllm22) — a **RDNA 4 (gfx12xx)** vLLM image with `mxfp4_16` kernel support. No other vLLM build currently loads these weights.
+
+---
+
+## 1. Introduction
+
+This is an **MXFP4-16** (Mixed-precision 4-bit with 16-element group size) quantized variant of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7), produced using compressed-tensors with an **IQ4_NL** codebook.
+
+The quantization scheme:
+- **Weight bits:** 4-bit per group of 16 elements
+- **Codebook:** IQ4_NL (Improved Q4 Normal) — 16 entries, asymmetric, FP4-like scale
+- **Target:** All `Linear` layers (MoE experts + FFN + attention projections)
+- **Excluded:** Attention `qkv_proj` scales, `block_sparse_moe.gate`, `lm_head`, `embed_tokens`, MTP layers, norms
+- **KV cache:** FP8 (e4m3), no dynamic quantization
+
+The result is a model that retains near-BF16 quality while fitting in significantly less VRAM, friendly to high-memory systems (128GB+ unified memory, multi-GPU 4×48 setups, RDNA4/GFX12xx GPUs).
+
+---
+
+## 2. Model Architecture
+
+MiniMax-M2.7 is a **456B-parameter sparse MoE** model with:
+- **456B total parameters** (sparse), **~30B activated** per token
+- **256 routed experts** per MoE layer, top-8 routing
+- **62 transformer layers**
+- **3 MTP (Multi-Token Prediction)** layers for speculative decoding
+- **200k context window**
+- Native tool-calling support
+
+Key architectural details from `config.json`:
+- `hidden_size`: 3072, `num_attention_heads`: 48, `num_key_value_heads`: 8, `head_dim`: 128
+- `num_local_experts`: 256, `num_experts_per_tok`: 8
+- `rope_theta`: 5,000,000, `max_position_embeddings`: 204,800
+
+---
+
+## 3. Quantization Details
+
+### What was quantized
+
+| Layer type | Quantization | Notes |
+|---|---|---|
+| MoE expert weights (w1/w3/w2) | MXFP4-16, IQ4_NL | Merged `w13_weight_packed` + scales |
+| FFN intermediate (gate/up/proj) | MXFP4-16, IQ4_NL | Standard linear layers |
+| Attention projections (qkv) | MXFP4-16, IQ4_NL | QKV split handled correctly |
+
+### What was NOT quantized
+
+| Layer | Reason |
+|---|---|
+| `self_attn.{k,v}_proj` scales | Per-tensor FP16 (no quantization) |
+| `block_sparse_moe.gate` | Router — kept BF16 |
+| `e_score_correction_bias` | MoE bias — kept BF16 |
+| `lm_head` | Output projection — kept BF16 |
+| `embed_tokens` | Embedding — kept BF16 |
+| MTP layers | Speculative decoding heads — kept BF16 |
+| RMSNorm layers | Normalizations — kept BF16 |
+| KV cache | FP8 (e4m3), calibrated scales |
+
+### KV Cache
+
+FP8 (e4m3) KV cache is used at runtime (`--kv-cache-dtype fp8_e4m3`). Per-layer scales are calibrated during quantization and stored alongside weights.
+
+---
+
+## 4. Runtime Requirements
+
+### Hardware
+
+- **GPU:** RDNA 4 (gfx12xx) — tested on 4× RX 9700 (RDNA4)
+- **Memory:** 128GB+ recommended for long-context workloads
+- **OS:** Linux with ROCm support
+
+### Docker Runtime
+
+The **only** validated way to run this model is with the prebuilt RDNA4 vLLM image:
+
+```bash
+# Pull the runtime image
+docker pull tcclaviger/vllm22:latest
+
+# Run with 8 GPUs
+./run-minimax-m2.7-mxfp416.sh <container_name> <port>
+```
+
+This image includes:
+- Custom Triton attention kernels tuned for RDNA4 (10× faster than ROCm attention at long context)
+- Fixed FP8 KV-cache quantization path (2× throughput improvement)
+- Tuned GEMM configs for RX 9700
+- MXFP4-16 kernels compiled for gfx12xx
+
+---
+
+## 5. Local Deployment
+
+### vLLM (Recommended)
+
+Using the RDNA4 Docker image:
+
+```bash
+vllm serve djdeniro/MiniMax-M2.7-MXFP416 \
+  --served-model-name minimax-m2.7-mxfp416 \
+  --tensor-parallel-size 8 \
+  --enable-expert-parallel \
+  --disable-cascade-attn \
+  --reasoning-parser minimax_m2 \
+  --enable-auto-tool-choice \
+  --tool-call-parser minimax_m2 \
+  --trust-remote-code \
+  --gpu-memory-utilization 0.93 \
+  --max-model-len 180000 \
+  --kv-cache-dtype fp8_e4m3 \
+  --attention-backend TRITON_ATTN \
+  --override-generation-config '{"max_tokens": 16384}'
+```
+
+Or with Docker:
+
+```bash
+docker run --name minimax-mxfp416 \
+  --rm --tty --ipc=host --shm-size=128g \
+  --device /dev/kfd:/dev/kfd \
+  --device /dev/dri/renderD128:/dev/dri/renderD128 \
+  --device /dev/dri/renderD129:/dev/dri/renderD129 \
+  --device /dev/dri/renderD130:/dev/dri/renderD130 \
+  --device /dev/dri/renderD132:/dev/dri/renderD132 \
+  --device /dev/dri/renderD137:/dev/dri/renderD137 \
+  --device /dev/dri/renderD138:/dev/dri/renderD138 \
+  --device /dev/dri/renderD139:/dev/dri/renderD139 \
+  --device /dev/dri/renderD140:/dev/dri/renderD140 \
+  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
+  -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
+  -e TRUST_REMOTE_CODE=1 \
+  -e PYTORCH_TUNABLEOP_ENABLED=1 \
+  -e PYTORCH_TUNABLEOP_TUNING=0 \
+  -p 8000:8000 \
+  tcclaviger/vllm22:latest \
+  bash -c "cp /patches/vllm22_minimax_m2.py /app/vllm/vllm/model_executor/models/minimax_m2.py && \
+    /app/.venv/bin/pip install -q sentencepiece && \
+    exec /app/.venv/bin/vllm serve \
+      /app/models/models/vllm/MiniMax-M2.7-MXFP416 \
+      --served-model-name minimax-m2.7-mxfp416 \
+      --host 0.0.0.0 --port 8000 \
+      --trust-remote-code \
+      --tensor-parallel-size 8 \
+      --disable-cascade-attn \
+      --reasoning-parser minimax_m2 \
+      --enable-auto-tool-choice --tool-call-parser minimax_m2 \
+      --enable-prefix-caching --gpu-memory-utilization 0.93 \
+      --max-model-len 180000 --max-num-seqs 48 --max-num-batched-tokens 2048 \
+      --kv-cache-dtype fp8_e4m3 \
+      --enable-expert-parallel \
+      --attention-backend TRITON_ATTN \
+      --override-generation-config '{\"max_tokens\": 16384}'"
+```
+
+### API Usage (OpenAI-compatible)
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY",
+)
+
+completion = client.chat.completions.create(
+    model="minimax-m2.7-mxfp416",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Explain what MXFP4 quantization is."}
+    ],
+    temperature=1.0,
+    max_tokens=1024,
+)
+print(completion.choices[0].message.content)
+```
+
+### Tool Calling
+
+MiniMax-M2.7 has native function calling support. Use `reasoning_parser=minimax_m2` and `tool_call_parser=minimax_m2`:
+
+```python
+messages = [
+    {"role": "user", "content": [
+        {"type": "text", "text": "What's the weather in Tokyo?"},
+    ]}
+]
+# The model will generate tool calls with the correct format
+```
+
+---
+
+## 6. Chat Template
+
+The model uses a custom Jinja chat template supporting:
+
+- **System messages** with dynamic tool injection
+- **Tool calls** in XML format (`<minimax:tool_call>` / `</minimax:tool_call>`)
+- **Reasoning content** (`<think>` / `</think>`)
+- **Tool responses** with `<response>` XML tags
+- **Generation prompts** with thinking prefix
+
+Example with `apply_chat_template`:
+
+```python
+from transformers import AutoProcessor, AutoModelForCausalLM
+
+processor = AutoProcessor.from_pretrained(
+    "djdeniro/MiniMax-M2.7-MXFP416",
+    trust_remote_code=True
+)
+model = AutoModelForCausalLM.from_pretrained(
+    "djdeniro/MiniMax-M2.7-MXFP416",
+    device_map="auto",
+    dtype="auto",
+    trust_remote_code=True
+)
+
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "Hello, how are you?"}
+]
+
+inputs = processor.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_dict=True,
+    return_tensors="pt",
+).to(model.device)
+
+generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+output = processor.decode(generated_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+print(output)
+```
+
+---
+
+## 7. Inference Parameters
+
+Recommended defaults:
+- `temperature`: 1.0
+- `top_p`: 0.95
+- `top_k`: 40
+- `max_tokens`: 16384 (configurable)
+
+---
+
+## 8. Acknowledgments
+
+- Base model: [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
+- Quantization inspiration: [tcclaviger/Step-3.7-Flash-240REAP-MXFP416](https://huggingface.co/tcclaviger/Step-3.7-Flash-240REAP-MXFP416)
+- Runtime: [tcclaviger/vllm22](https://hub.docker.com/r/tcclaviger/vllm22)
+
+---
+
+## 9. License
+
+This quantized variant inherits the [Apache 2.0 license](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE) from the base model.