---
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/LICENSE
thumbnail: https://huggingface.co/AlexAtomic/qwen35-4b-GGUF/resolve/main/hero.png
base_model:
- Qwen/Qwen3.5-4B
base_model_relation: quantized
quantized_by: AlexAtomic
pipeline_tag: text-generation
library_name: gguf
tags:
- atomic-chat
- qwen
- qwen3
- gguf
- imatrix
- quantized
- llama.cpp
---
**Qwen3.5 4B**, self-quantized to GGUF by [Atomic Chat](https://atomic.chat). Built straight from Qwen's original weights with a per-tensor importance matrix. Runs fully offline.
## Highlights
- **Efficient hybrid architecture** combining Gated Delta Networks with sparse Mixture-of-Experts for high-throughput inference at low latency and cost.
- **Unified vision-language foundation** trained with early fusion on multimodal tokens (these GGUF quants cover the text path).
- **262,144-token native context**, extensible up to ~1,010,000 tokens.
- **Global linguistic coverage**: Qwen reports support for 201 languages and dialects.
- **Thinking and instruct modes**, each with its own recommended sampling presets.
- **Full quant ladder** with an importance matrix on every quant over [`calibration_datav3`](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8).
> [!NOTE]
> These GGUFs are **self-quantized from the original weights**, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model.
> [!IMPORTANT]
> Always pass `--jinja` so the **Qwen3.5 4B chat template** is applied. Without it the model can emit malformed turns.
## Model Overview
| Property | Value |
|---|---|
| Base model | `Qwen/Qwen3.5-4B` |
| Total parameters | 4B |
| Layers | 32 |
| Context length | 262,144 native, extensible up to ~1,010,000 |
| Architecture | Causal LM with vision encoder; hybrid Gated DeltaNet + Gated Attention, hidden dim 2560, trained with MTP |
| This repo | GGUF quants (imatrix), text path |
Scores are Qwen's published results for the base `Qwen/Qwen3.5-4B`. Quantization preserves the large majority of this; `Q4_K_M` and up sit within a point or two of full precision.
## Choosing a quant
| Quant | Size | Notes |
|---|---|---|
| `Q2_K` | 1.9 GB | Smallest. Minimal RAM, clear quality drop. |
| `IQ3_M` | 2.2 GB | Beats Q3 at similar size thanks to imatrix. Best low-RAM pick. |
| `Q3_K_M` | 2.3 GB | Low quality but usable. |
| `Q3_K_L` | 2.4 GB | A step above Q3_K_M. |
| `IQ4_XS` | 2.5 GB | Excellent quality for size. Recommended low-bit. |
| `Q4_K_S` | 2.6 GB | Compact Q4, fast. |
| **`Q4_K_M`** | 2.7 GB | **Recommended default. Best balance of size, speed and quality.** |
| **`UD-Q4_K_XL`** | 2.9 GB | **Dynamic. Embeddings and output kept at Q8_0 for higher quality at a Q4 footprint.** |
| `Q5_K_S` | 3.0 GB | Higher quality. |
| `Q5_K_M` | 3.1 GB | Higher quality, low loss. |
| `Q6_K` | 3.5 GB | Near lossless. |
| `Q8_0` | 4.5 GB | Effectively lossless, reference quality. |
> [!TIP]
> Pick the largest file that fits your (V)RAM with room for context. `Q4_K_M` or `UD-Q4_K_XL` is the sweet spot for most setups; `Q6_K` or `Q8_0` for maximum fidelity.
## Get started
Run Qwen3.5 4B locally with:
- **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AlexAtomic/qwen35-4b-GGUF`, pick a quant, hit **Use this model**.
- **llama.cpp:** `llama-server -hf AlexAtomic/qwen35-4b-GGUF:Q4_K_M --jinja -c 8192`
- **Ollama:** `ollama run hf.co/AlexAtomic/qwen35-4b-GGUF:Q4_K_M`
- **LM Studio / Jan:** search the repo id, download any quant.
## Best practices
| Parameter | Value |
|---|---|
| temperature | 0.7 |
| top_p | 0.8 |
| top_k | 20 |
| min_p | 0.0 |
| presence_penalty | 1.5 |
| repetition_penalty | 1.0 |
Qwen's recommended Instruct (non-thinking) settings. Thinking mode for general tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.
## Run in llama.cpp
```bash
git clone https://github.com/ggerganov/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server
```
```bash
./llama.cpp/build/bin/llama-server \
-hf AlexAtomic/qwen35-4b-GGUF:UD-Q4_K_XL \
--jinja -ngl 99 -c 8192 -fa on
```
## How these were made
1. Download `Qwen/Qwen3.5-4B` (original weights).
2. Convert to f16 GGUF with [llama.cpp](https://github.com/ggerganov/llama.cpp).
3. Build an importance matrix over `calibration_datav3` (100 chunks).
4. Quantize the full ladder with `--imatrix`.
5. `UD-Q4_K_XL` additionally pins the token-embedding and output tensors to `Q8_0`.
## License
Released by Qwen under the Apache 2.0 license. Quantized by Atomic Chat.
