---
pipeline_tag: image-text-to-text
license: other
license_name: minimax-community
license_link: LICENSE
library_name: transformers
base_model: MiniMaxAI/MiniMax-M3
tags:
  - multimodal
  - moe
  - fp8
  - block-fp8
  - sglang
  - rocm
---

# MiniMax-M3 — MoE-only block-FP8 (128×128)

Block-FP8 quantization of [MiniMaxAI/MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3)
applied **only to the MoE expert weights**. Everything else stays BF16.

- **408 GB** vs 796 GiB for the BF16 original (~102 GB/GPU at TP4, vs 199 GB)
- Quantized **directly from BF16 in a single rounding step**
- Vision tower intact — still the full multimodal `MiniMaxM3SparseForConditionalGeneration`

## What is and isn't quantized

| Quantized (block-FP8 128×128) | Left in BF16 / FP32 |
| --- | --- |
| routed experts `w1` / `w2` / `w3` | all attention, incl. `index_q_proj` / `index_k_proj` (sparse indexer) |
| shared experts `gate/up/down_proj` | router `gate` (FP32 — precision-critical for expert selection) |
| | dense MLP layers 0–2 |
| | vision tower, `lm_head`, `embed_tokens` |

Routed experts are ~96.6% of the parameters, so MoE-only quantization captures
essentially all of the memory saving while leaving the accuracy-sensitive and
long-context-critical paths untouched. Shared experts are included because they
run for *every* token (unlike top-4-of-128 routed experts) and account for ~1/5
of MoE decode traffic.

22,059 tensors were quantized (21,888 routed = 57 layers × 128 experts × 3, plus
171 shared = 57 × 3).

## Format

`weight_scale_inv` follows the DeepSeek-V3 block-FP8 contract — despite the name
it is a **multiplier**:

```
dequantized = fp8_weight * weight_scale_inv
```

Weights are `float8_e4m3fn`; scales are FP32 with shape
`[ceil(N/128), ceil(K/128)]`. Activations are quantized dynamically in **1×128**
groups (per token, along K) — hence `activation_scheme: dynamic`, since
block-wise FP8 cannot use static activation scales.

Mean round-trip error vs the original BF16 is **~2.4%**.

## Usage (SGLang)

```bash
python -m sglang.launch_server \
  --model-path Model-SafeTensors/MiniMax-M3-MoE-FP8-128x128 \
  --trust-remote-code \
  --tp-size 4 \
  --moe-runner-backend triton \
  --context-length 98304
```

Requires an SGLang build with MiniMax-M3 support (merged 2026-07-11).

### ⚠️ Known serving issues on AMD / ROCm

Both are properties of the serving stack, **not** of these weights. Both produce
fluent-looking garbage with no error, so smoke-test before trusting output.

1. **Do not use `--moe-runner-backend aiter` with MiniMax-M3.** M3's MoE uses a
   clamped SwiGLU (`gemm1_alpha=1.702`, `gemm1_clamp_limit=7.0`):

   ```
   gate = gate.clamp(max=limit); up = up.clamp(-limit, limit)
   out  = gate * sigmoid(gate * alpha) * (up + 1)
   ```

   aiter's `fused_moe` has no alpha argument and its `ActivationType` offers only
   `{No, Gelu, Silu, Swiglu}`, so the runner silently computes a plain SiLU.
   This affects **BF16 too**, not just FP8.

2. **`SGLANG_USE_AITER=1` together with `--moe-runner-backend triton` can
   corrupt output.** On builds where the block-FP8 post-load path gates its aiter
   `(16,16)` weight pre-shuffle on the `SGLANG_USE_AITER` env var rather than on
   the selected MoE runner, weights are shuffled for aiter and then consumed
   un-shuffled by triton. Workaround: set `SGLANG_USE_AITER=0`, or use a build
   whose plain-FP8 branch gates the shuffle on the runner (the
   `mxfp8 → block-fp8` branch already does).

## Why MoE-only, and why 128×128

- **128×128 blocks** are the granularity that reaches the optimized block-FP8
  kernels (DeepSeek-V3 style), rather than per-tensor or per-channel scaling.
- **Single rounding.** An MXFP8 checkpoint served on gfx942 (CDNA3, which has no
  MX matmul hardware) is converted to block-FP8 at load — a dequant→requant round
  trip that drops from 1×32 scales to 128×128, a 512× reduction in scale count,
  on top of already-quantized values. Quantizing straight from BF16 avoids that
  second rounding.
- **BF16 attention** keeps the sparse-attention indexer exact, which matters most
  at long context where attention dominates the decode step.

## Provenance

Quantized from the official BF16 release using the DeepSeek-V3 block-FP8 formula
(`amax/448` per 128×128 tile, `clamp(min=1e-4)`), verified **bit-identical** to
SGLang's own `bf16_to_block_fp8_128` reference implementation.

## Evaluation

**Not formally evaluated.** Validated by generation smoke tests and the ~2.4%
weight round-trip error only — no GSM8K / MMLU / perplexity numbers were
produced. Benchmark before production use.

## License

Inherits the MiniMax Model License from the base model. See `LICENSE`.