Qwen3-Omni-30B-A3B-Instruct NVFP4 (W4A8)
Pre-quantized NVFP4 version of Qwen/Qwen3-Omni-30B-A3B-Instruct for deployment on NVIDIA Blackwell GPUs (RTX 5090, B100, B200).
Key Features
- 4-bit weight, 8-bit activation (W4A8) quantization using NVIDIA ModelOpt NVFP4
- Pre-quantized checkpoint — weights are packed as FP4 (uint8), not online quantization
- ~3.6x compression on MoE expert weights (54 GB → 15 GB)
- Same capabilities as the original: text, image, audio, video input → text + speech output
Quantization Details
| Component | Precision | Quantized? |
|---|---|---|
| Thinker MoE experts (gate_up_proj, down_proj) | NVFP4 packed uint8 | Yes — pre-quantized |
| Thinker attention (q/k/v/o_proj) | NVFP4 (calibrated) | Yes |
| Thinker lm_head | BF16 | No |
| Thinker MoE router gates | BF16 | No |
| Audio Encoder | BF16 | No |
| Vision Encoder | BF16 | No |
| Talker (MoE) | BF16 | No |
| Code2Wav | BF16 | No |
Weight Format
MoE expert weights are stored as:
gate_up_proj: packed uint8 (2× FP4 values per byte)gate_up_proj_scale: float8_e4m3fn (per-block-of-16 FP8 scale)gate_up_proj_scale_2: bfloat16 (global per-tensor scale)- Same pattern for
down_proj
Quantization Config
{
"quant_algo": "NVFP4",
"quant_method": "modelopt",
"group_size": 16,
"weights": {"num_bits": 4, "type": "float"},
"input_activations": {"num_bits": 8, "type": "float"},
"kv_cache": {"num_bits": 8, "type": "float"}
}
Memory Comparison
| Config | Checkpoint Size | Notes |
|---|---|---|
| BF16 (original) | ~60 GB | Full precision |
| FP8 (ModelOpt) | ~40 GB | Online quantization |
| NVFP4 (this model) | ~27 GB | Pre-quantized FP4 experts |
Hardware Requirements
| Purpose | Hardware | Notes |
|---|---|---|
| Inference | RTX 5090, B100, B200 (Blackwell SM 100+) | Native FP4 hardware support required |
| Inference (not supported) | RTX 4090, A100, H100 | No FP4 hardware — use FP8 instead |
Quantization Methodology
- Base model: Qwen/Qwen3-Omni-30B-A3B-Instruct (BF16)
- Tool: NVIDIA ModelOpt v0.42.0
- Method: NVFP4 (W4A8) with static calibration (256 diverse prompts)
- Calibration: 256 English + Chinese prompts covering factual, creative, code, math, and reasoning tasks
- Attention layers: Quantized via
mtq.quantizewith calibration - Expert layers: Manually packed via
NVFP4QTensor.quantize()(fused expert tensors not supported bymtq.quantize) - Excluded: lm_head, MoE router gates, all non-thinker stages
Original Model
This is a quantized version of Qwen/Qwen3-Omni-30B-A3B-Instruct. Please refer to the original model card for full capabilities, benchmarks, and usage documentation.
License
Apache License 2.0 (same as the original model)
- Downloads last month
- 476
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4
Base model
Qwen/Qwen3-Omni-30B-A3B-Instruct
# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4") model = AutoModelForMultimodalLM.from_pretrained("shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4", device_map="auto")