metadata
license: mit
language:
- en
- zh
tags:
- custom
- int4
- gptq
- quantized
- amd
- rdna4
- rocm
base_model: Qwen/Qwen3-14B
pipeline_tag: text-generation
library_name: custom
Qwen3-14B INT4 Mixed (GPTQ + Hadamard)
Mixed INT4/INT8 quantization of Qwen/Qwen3-14B using custom quantization pipeline.
Best quality — sensitive layers kept at INT8 based on Hessian-weighted sensitivity analysis.
Quality
| Metric | This Model | FP16 Reference | llama.cpp Q4_K_M |
|---|---|---|---|
| Perplexity (WikiText-2) | 7.692 | ~7.5 | 7.715 |
| ARC-Challenge (250) | 92.8% | ~94% | 90.8% |
| MMLU (250, 14 subjects) | 75.6% | ~78% | 72.8% |
Performance (AMD Radeon AI PRO R9700)
| Metric | Speed |
|---|---|
| Decode (ctx=128) | 61 t/s |
| Prefill (pp512) | 2076 t/s |
| VRAM | 9.9 GB |
Quantization Details
- Method: INT4 asymmetric with Hadamard rotation + GPTQ calibration
- Sensitive layers: INT8 (top 20% by Hessian-weighted error)
- Block size: 32
- Calibration: 256 samples × 256 tokens from RedPajama
- KV cache: FP8 E4M3 at inference time
File Format
Custom .pt format — requires rdna4-quant engine.
embed.pt — embedding weights (FP16)
layer_000.pt — layer 0 (quantized weights + scales + metadata)
...
layer_039.pt — layer 39
final_norm.pt — final RMSNorm
lm_head.pt — LM head (FP16)
meta.pt — quantization metadata
Usage
git clone https://github.com/JohnTDI-cpu/rdna4-quant
cd rdna4-quant
pip install -r requirements.txt
# Download weights
huggingface-cli download JohnTdi/Qwen3-14B-INT4-Mixed-GPTQ --local-dir quantized_v4_gptq
# Run inference
python int4_engine_v5.py --quant-dir quantized_v4_gptq --chat
# Or start API server
python api_server.py --quant-dir quantized_v4_gptq
Hardware Requirements
- AMD GPU with ROCm 6.x+ support (RDNA3/4, MI300X)
- ~10 GB VRAM
- ROCm 6.x or 7.x, PyTorch 2.6+
License
MIT — same as the inference engine. Base model (Qwen3-14B) has its own license from Alibaba.