---
pipeline_tag: text-generation
license: apache-2.0
library_name: vllm
tags:
- gptq
- 4-bit
- quantization
- hunyuan
- hy3
- moe
- rtn
- vllm
- text-generation
base_model:
- tencent/Hy3
---

# Tencent Hy3 GPTQ RTN 4bit tp8 (MPT layer removed)

This repository contains a **4-bit RTN-quantized** version of the [Tencent Hy3](https://huggingface.co/tencent/Hy3) Mixture-of-Experts model.

Hy3 is a 295B-parameter MoE model with 21B active parameters, 192 experts (top-8 routed), and a 256K context window. Quantizing it to 4-bit brings the weight footprint down to ~150 GB, making it possible to serve the **full 256K context** on a single 8x24 GB node with KVarN KV cache compression.

## ⚠️ Required vLLM Branch

This model requires a patched vLLM build. Stock vLLM cannot load it due to several bugs in weight-name remapping, dynamic GPTQ MoE group-size handling, and TurboQuant KV cache shaping:

➡️ **[avtc/vllm @ fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26](https://github.com/avtc/vllm/tree/fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26)**

The branch includes these fixes:

| Commit | Fix |
|:---|:---|
| `a1ca5aff6` | Remap `e_score_correction_bias`→`expert_bias` for transformers 5.x checkpoint renames |
| `80437c75e` | Respect dynamic `group_size=64` override for GPTQ MoE Marlin support + WNA16 fallback |
| `c2905928f` | Fix `shared_mlp` weight prefix so shared-expert weights load correctly |
| `4e3126f5f` | Use `WeightsMapper` for HY3 transformers weight renames + quant detection |
| `325a5ddd7` | Disable TurboQuant boundary skip layers (mixed-backend KV cache allocation crash) |
| `33a482b71` | Fix `layer_cache_dtype_str` for TurboQuant in KV cache reshape (`TQFullAttentionSpec` has `kv_quant_mode=NONE`) |
+ fixes to allow run AWQ/GPTQ quants of Mimo-v2.5
+ chery picked commit: [Attention] Add KVarN: calibration-free variance-normalized KV-cache quantization backend
  from https://github.com/vllm-project/vllm/pull/46812

Build from source:

```bash
git clone -b fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26 https://github.com/avtc/vllm.git
cd vllm
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
TORCH_CUDA_ARCH_LIST="8.6" uv pip install --editable . --torch-backend=auto
```

> Set `TORCH_CUDA_ARCH_LIST` to your GPU's compute capability to skip unused architectures and speed up the build (e.g. `8.6` for RTX 3090, `8.9` for RTX 4090, `9.0` for H100).

## Quantization Details

Quantized with **[GPTQModel](https://github.com/ModelCloud/GPTQModel)** (`7.2.0`) using the **RTN** (Round-To-Nearest) fallback strategy.

| Property | Value |
|:---|:---|
| Quant method | GPTQ (RTN fallback) |
| Bits | 4 |
| Symmetric | `true` |
| `desc_act` | `false` |
| Global group size | 128 (dense layers, attention projections) |
| Expert / shared-MLP group size | **64** (dynamic override) |
| Fallback threshold | 0.5% |
| Pack dtype | int32 |

## Verified Hardware & Performance

This model is verified to run with Tensor Parallel on **8x NVIDIA RTX 3090** (24 GB each) GPUs with the **full 256K (262,144) context window**, using kvarn_k4v2_g128 4-bit K, 2-bit V near lossless cache compression.

- Tensor parallel: **8** (no expert parallel — expert group size 64 divides cleanly across 8 GPUs)
- KV cache: `kvarn_k4v2_g128`
- Max model len: **262,144**
- Model weight footprint: ~150 GB
- Reasoning (`<think>`) and tool-calling work.

## Quick Start

```bash
export VLLM_SLEEP_WHEN_IDLE=1
export TORCH_CUDA_ARCH_LIST="8.6"
export CUDA_VISIBLE_DEVICES=4,5,6,7,0,1,2,3
export RAY_memory_monitor_refresh_ms=0
export NCCL_CUMEM_ENABLE=0
export VLLM_ENABLE_CUDAGRAPH_GC=1

export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export VLLM_FLOAT32_MATMUL_PRECISION=high
export OMP_NUM_THREADS=1

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve /path/to/tencent-Hy3-RTN \
  -tp 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3 \
  --port 8000 \
  --host 0.0.0.0 \
  --uvicorn-log-level info \
  --trust-remote-code \
  --max-num-seqs 4 \
  --seed 1234 \
  --max-model-len auto \
  --skip-mm-profiling \
  -O3 \
  --no-use-tqdm-on-load \
  --default-chat-template-kwargs '{"interleaved_thinking": true}' \
  --performance-mode balanced \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --kv-cache-dtype kvarn_k4v2_g128 \
  --gpu-memory-utilization 0.945
```

Then call the OpenAI-compatible API:

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy3",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    # reasoning_effort: "no_think" (default, direct response),
    #                   "low", "high" (deep chain-of-thought)
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)
```

**Recommended sampling parameters:** `temperature=0.9`, `top_p=1.0`.

**Reasoning mode:** Set `reasoning_effort` to `"high"` for complex tasks (math, coding, reasoning), `"low"` for faster thinking or `"no_think"` for direct responses.

## Verified Functionality

- ✅ Functional 2D HTML aquarium generation (without reasoning enabled) — working artifact.
- ✅ Reasoning mode (`<think>`) engages and produces output.

## Original Model

For full details on the architecture, benchmarks, and capabilities of the base model, see the original model card:
➡️ **[tencent/Hy3](https://huggingface.co/tencent/Hy3)**

| Property | Value |
|:---|:---|
| Architecture | Mixture-of-Experts (MoE) |
| Total Parameters | 295B |
| Activated Parameters | 21B |
| Number of Layers | 80 (MTP layer was removed during quantization) |
| Attention Heads | 64 (GQA, 8 KV heads, head dim 128) |
| Hidden Size | 4096 |
| Intermediate Size | 13312 (dense), 1536 (expert) |
| Context Length | 256K |
| Vocabulary Size | 120,832 |
| Number of Experts | 192, top-8 activated |
| License | Apache 2.0 |

## Acknowledgments

- **[Tencent Hunyuan Team](https://github.com/Tencent-Hunyuan/Hy3)** for the original Hy3 model.
- **[GPTQModel](https://github.com/ModelCloud/GPTQModel)** team for the quantization toolkit.
- **[vLLM](https://github.com/vllm-project/vllm)** team for the inference engine.
- **[KVarN](https://github.com/huawei-csl/KVarN)** team for abilty to use large near-lossless KV cache

## License

This model inherits the **Apache License 2.0** of the base model. See the [original license](https://huggingface.co/tencent/Hy3/blob/main/LICENSE).

## Attribution
```
@misc{muller2026kvarn,
      title={KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks}, 
      author={Lorenz K. Muller and Philippe Bich and Chiara Boretti and Hyun-Min Chang and Jiawei Zhuang and Lukas Cavigelli},
      year={2026},
      eprint={2606.03458},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={http://arxiv.org/abs/2606.03458}
}
```