---
license: apache-2.0
base_model: Qwen/Qwen3.5-9B
language:
- en
- zh
- ko
- ja
tags:
- hlwq
- quantized
- compressed-tensors
- int4
- marlin
- vllm
pipeline_tag: text-generation
arxiv: '2603.29078'
library_name: transformers
---

> [!IMPORTANT]
> **Naming notice (2026-04-10).** The "HLWQ" technique used in this model is being rebranded to **HLWQ (Hadamard-Lloyd Weight Quantization)**. The change is only the name; the algorithm and the weights in this repository are unchanged.
>
> The rebrand resolves a name collision with an unrelated, earlier KV cache quantization method also named HLWQ ([Han et al., arXiv:2502.02617, 2025](https://arxiv.org/abs/2502.02617)). HLWQ addresses **weight** quantization with a **deterministic Walsh-Hadamard rotation** and Lloyd-Max scalar codebook; Han et al.'s HLWQ addresses **KV cache** quantization with a **random polar rotation**. The two methods are technically distinct.
>
> Existing loaders that load this repository by ID continue to work without changes. Future model uploads will use the HLWQ name.
>
> Reference paper for this technique: [arXiv:2603.29078](https://arxiv.org/abs/2603.29078) (v2 in preparation; v1 still uses the old name).

# Qwen3.5-9B — HLWQ INT4

**Native vLLM. Marlin kernel. Zero plugin.**

HLWQ Q5 preprocessing produces **better INT4 weights** than direct quantization — stored in CompressedTensors format for native vLLM inference.

## Quick Start — vLLM (one command)

```bash
pip install vllm
vllm serve caiovicentino1/Qwen3.5-9B-HLWQ-Q5 --language-model-only --enforce-eager
```

That's it. No plugin, no `pip install polarquant`, no custom code.

**Tested results:**

| GPU | tok/s |
|-----|-------|
| A100 80GB | **168 tok/s** (9B) |
| RTX PRO 6000 96GB | **44 tok/s** (9B) / **18 tok/s** (27B) |

## Quick Start — HuggingFace Transformers

```bash
pip install polarquant
```

```python
import polarengine_vllm  # auto-registers with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("caiovicentino1/Qwen3.5-9B-HLWQ-Q5", device_map="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("caiovicentino1/Qwen3.5-9B-HLWQ-Q5", trust_remote_code=True)

inputs = tokenizer("Hello!", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```


## Consumer GPU Compatibility

| GPU | VRAM | Works? | Expected tok/s |
|-----|------|--------|---------------|
| RTX 4060 | 8 GB | YES | ~20 |
| RTX 3060/4070 | 12 GB | YES | ~30 |
| RTX 4080 | 16 GB | YES | ~35 |
| RTX 4090 | 24 GB | YES | ~40 |
| A100 | 80 GB | YES | ~168 |

## Why HLWQ INT4 is Better

Standard INT4 (GPTQ/AWQ) quantizes weights directly — outliers cause errors.

HLWQ adds a **preprocessing step**:

1. **Hadamard rotation** — distributes weight energy uniformly (eliminates outliers)
2. **Lloyd-Max Q5** — MSE-optimal quantization for the resulting Gaussian distribution
3. **Dequant → INT4** — the cleaned weights produce better INT4 than direct quantization

| Method | PPL (lower = better) |
|--------|---------------------|
| BF16 baseline | 6.37 |
| **HLWQ → INT4** | **6.56** |
| Direct INT4 | 6.68 |

**Same speed as GPTQ/AWQ, better quality.**

## Important Flags

| Flag | Why |
|------|-----|
| `--language-model-only` | Qwen3.5 is multimodal — this skips the vision encoder (we only quantized text) |
| `--enforce-eager` | Required on Blackwell GPUs (cc 12.0). Optional on A100/H100 (faster without it) |

## Links

- Paper: [arxiv.org/abs/2603.29078](https://arxiv.org/abs/2603.29078)
- GitHub: [github.com/caiovicentino/polarengine-vllm](https://github.com/caiovicentino/polarengine-vllm)
- PyPI: `pip install polarquant`
- Base model: [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)