Hugging Face's logo

Vishva007
/
Qwen3.8-4B-Distill-W4A16-AutoRound-GPTQ

Image-Text-to-Text
Safetensors
qwen3_5
quantization
auto-round
gptq
llm-compressor
compressed-tensors
4bit
text-generation
reasoning
distillation
conversational
4-bit precision
Model card Files
xet
 Community
Qwen3.8-4B-Distill-W4A16-AutoRound-GPTQ
File size: 4,897 Bytes
5684aab
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e326560
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5684aab
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
---
license: apache-2.0
base_model: empero-ai/Qwen3.8-4B-Distill
tags:
- quantization
- auto-round
- gptq
- llm-compressor
- compressed-tensors
- 4bit
- text-generation
- image-text-to-text
- reasoning
- distillation
pipeline_tag: image-text-to-text
---

# Qwen3.8-4B-Distill (W4A16 Quantized via AutoRound)

This repository contains a **W4A16 (4-bit weights, 16-bit activations)** quantized version of [empero-ai/Qwen3.8-4B-Distill](https://huggingface.co/empero-ai/Qwen3.8-4B-Distill), quantized using Intel's [AutoRound](https://github.com/intel/auto-round) algorithm.

---

## ⚡ Quantization Details

Calibrated and quantized with fine-grained group sizes and high iteration depth to preserve reasoning traces (`<think>` blocks) and multimodal capabilities:

* **Algorithm:** [Intel AutoRound](https://github.com/intel/auto-round)
* **Precision / Scheme:** W4A16 (4-bit weights, 16-bit activations)
* **Group Size:** 32 (fine-grained reconstruction fidelity)
* **Symmetric (`sym`):** `True`
* **Calibration Samples (`nsamples`):** 512
* **Sequence Length (`seqlen`):** 4096
* **Tuning Iterations (`iters`):** 1000 (Production-grade accuracy)
* **Vision Tower (`quant_nontext_module`):** `False` (Kept in **BF16** to preserve visual reasoning and OCR precision)
* **Special Modules (`layer_config`):** Multi-Token Prediction (`mtp`, `mtp.fc`) kept in native **bfloat16**

---

## 📦 Available Formats

Depending on your inference engine, choose the appropriate repository:

* **AutoRound Format:** [`Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound`](https://huggingface.co/Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound)
* **AutoGPTQ Format:** [`Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-GPTQ`](https://huggingface.co/Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-GPTQ)
* **LLM-Compressor / Compressed-Tensors Format:** [`Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-LLM-Compressor`](https://huggingface.co/Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-LLM-Compressor)

---

## 🚀 Usage & Quickstart

### 1. High-Throughput Serving via vLLM

```bash
# Using the GPTQ build
vllm serve Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-GPTQ \
    --dtype bfloat16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

```

---

## 📊 VRAM & Performance Benefits

* **Original Model (BF16):** ~8–10 GB VRAM required for full context inference
* **Quantized Model (W4A16 Group 32):** ~2.5–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs, laptops, and edge devices)
* **Throughput:** Lowers memory bandwidth pressure, accelerating token generation speeds during extended chain-of-thought (`<think>`) reasoning.

---


### 🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or Quantization.

#### PyTorch 2.13

| Template | CUDA Version | Docker Image | Template ID | Deploy |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch 2.13 (CUDA 12.6)** | 12.6 | `vishva123/cuda-12.6-pytorch-2.13-runpod` | `gmlupxnxfk` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=gmlupxnxfk) |
| **PyTorch 2.13 (CUDA 13.0)** | 13.0 | `vishva123/cuda-13.0-pytorch-2.13-runpod` | `y3j8xvk4f4` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=y3j8xvk4f4) |
| **PyTorch 2.13 (CUDA 13.2)** | 13.2 | `vishva123/cuda-13.2-pytorch-2.13-runpod` | `vigpissn5w` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=vigpissn5w) |

#### PyTorch 2.12

| Template | CUDA Version | Docker Image | Template ID | Deploy |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch 2.12 (CUDA 12.6)** | 12.6 | `vishva123/cuda-12.6-pytorch-2.12-runpod` | `ctmz86zmf0` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=ctmz86zmf0) |
| **PyTorch 2.12 (CUDA 13.0)** | 13.0 | `vishva123/cuda-13.0-pytorch-2.12-runpod` | `qjko5yiwzi` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=qjko5yiwzi) |
| **PyTorch 2.12 (CUDA 13.2)** | 13.2 | `vishva123/cuda-13.2-pytorch-2.12-runpod` | `ifg6xmye0f` | [![Deploy to RunPod](https://img.shields.io/badge/RunPod-Deploy%20Pod-purple?logo=runpod&logoColor=white)](https://runpod.io/console/deploy?template=ifg6xmye0f) |


## 📚 Acknowledgments

* **Original Distilled Model:** Developed by [Empero AI](https://huggingface.co/empero-ai/Qwen3.8-4B-Distill)
* **Base Architecture:** [Qwen Team](https://github.com/QwenLM/Qwen) (Alibaba)
* **Quantization Framework:** [Intel AutoRound](https://github.com/intel/auto-round)