---
license: mit
language:
- en
- zh
tags:
- custom
- int4
- gptq
- quantized
- amd
- rdna4
- rocm
base_model: Qwen/Qwen3-14B
pipeline_tag: text-generation
library_name: custom
---

# Qwen3-14B INT4 Mixed (GPTQ + Hadamard)

Mixed INT4/INT8 quantization of [Qwen/Qwen3-14B](https://huggingface.co/Qwen/Qwen3-14B) using custom quantization pipeline.

**Best quality** — sensitive layers kept at INT8 based on Hessian-weighted sensitivity analysis.

## Quality

| Metric | This Model | FP16 Reference | llama.cpp Q4_K_M |
|--------|-----------|----------------|-----------------|
| **Perplexity** (WikiText-2) | **7.692** | ~7.5 | 7.715 |
| **ARC-Challenge** (250) | **92.8%** | ~94% | 90.8% |
| **MMLU** (250, 14 subjects) | **75.6%** | ~78% | 72.8% |

## Performance (AMD Radeon AI PRO R9700)

| Metric | Speed |
|--------|-------|
| Decode (ctx=128) | **61 t/s** |
| Prefill (pp512) | **2076 t/s** |
| VRAM | **9.9 GB** |

## Quantization Details

- **Method**: INT4 asymmetric with Hadamard rotation + GPTQ calibration
- **Sensitive layers**: INT8 (top 20% by Hessian-weighted error)
- **Block size**: 32
- **Calibration**: 256 samples × 256 tokens from RedPajama
- **KV cache**: FP8 E4M3 at inference time

## File Format

Custom `.pt` format — requires [rdna4-quant](https://github.com/JohnTDI-cpu/rdna4-quant) engine.

```
embed.pt          — embedding weights (FP16)
layer_000.pt      — layer 0 (quantized weights + scales + metadata)
...
layer_039.pt      — layer 39
final_norm.pt     — final RMSNorm
lm_head.pt        — LM head (FP16)
meta.pt           — quantization metadata
```

## Usage

```bash
git clone https://github.com/JohnTDI-cpu/rdna4-quant
cd rdna4-quant
pip install -r requirements.txt

# Download weights
huggingface-cli download JohnTdi/Qwen3-14B-INT4-Mixed-GPTQ --local-dir quantized_v4_gptq

# Run inference
python int4_engine_v5.py --quant-dir quantized_v4_gptq --chat

# Or start API server
python api_server.py --quant-dir quantized_v4_gptq
```

## Hardware Requirements

- AMD GPU with ROCm 6.x+ support (RDNA3/4, MI300X)
- ~10 GB VRAM
- ROCm 6.x or 7.x, PyTorch 2.6+

## License

MIT — same as the inference engine. Base model (Qwen3-14B) has its own license from Alibaba.