---
tags:
- techwithsergiu
- gguf
library_name: gguf
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.5-0.8B/blob/main/LICENSE
pipeline_tag: text-generation
base_model:
- techwithsergiu/Qwen3.5-text-0.8B
---

# Qwen3.5-text-0.8B-GGUF

<img width="400px" src="https://qianwen-res.oss-accelerate.aliyuncs.com/logo_qwen3.5.png">

GGUF quants of [techwithsergiu/Qwen3.5-text-0.8B](https://huggingface.co/techwithsergiu/Qwen3.5-text-0.8B) —
the text-only bf16 derivative of [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B).

The visual tower has been removed before conversion. All text-backbone weights are
**identical** to the original — no retraining, no weight changes, no quality loss for
text tasks.

## Quants

| File | Type | Size | Notes |
|---|---|---|---|
| `Qwen3.5-text-0.8B-Q8_0.gguf` | Q8_0 | ~53% of f16 | near-lossless — for high-quality inference |
| `Qwen3.5-text-0.8B-Q6_K.gguf` | Q6_K | ~41% of f16 | excellent quality, good balance with f16 |
| `Qwen3.5-text-0.8B-Q5_K_M.gguf` | Q5_K_M | ~37% of f16 | very good quality, smaller than Q6 |
| `Qwen3.5-text-0.8B-Q4_K_M.gguf` | Q4_K_M | ~31% of f16 | ✅ recommended — best size/quality balance |
| `Qwen3.5-text-0.8B-Q4_K_S.gguf` | Q4_K_S | ~30% of f16 | optional — slightly smaller, slightly lower quality |

## Model family

![](diagrams/diagram_01.png)

| Model | Type | Base model |
|---|---|---|
| [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) | f16 · VLM · source | — |
| [techwithsergiu/Qwen3.5-0.8B-bnb-4bit](https://huggingface.co/techwithsergiu/Qwen3.5-0.8B-bnb-4bit) | BNB NF4 · VLM | Qwen/Qwen3.5-0.8B |
| [techwithsergiu/Qwen3.5-text-0.8B](https://huggingface.co/techwithsergiu/Qwen3.5-text-0.8B) | bf16 · text-only | Qwen/Qwen3.5-0.8B |
| [techwithsergiu/Qwen3.5-text-0.8B-bnb-4bit](https://huggingface.co/techwithsergiu/Qwen3.5-text-0.8B-bnb-4bit) | BNB NF4 · text-only | Qwen3.5-text-0.8B |
| **[techwithsergiu/Qwen3.5-text-0.8B-GGUF](https://huggingface.co/techwithsergiu/Qwen3.5-text-0.8B-GGUF)** | GGUF quants | Qwen3.5-text-0.8B |

The GGUF repo is derived from the text-only f16 model — same weights, different container
format. `base_model` points to the f16 text variant to keep the VLM and text lineages
distinct on the Hub.

## Inference

### llama.cpp

```bash
./llama.cpp/build/bin/llama-cli \
    -m Qwen3.5-text-0.8B-Q4_K_M.gguf \
    -p "What is the capital of Romania?" \
    -n 256
```

### LM Studio

Load any `.gguf` file from this repo directly in [LM Studio](https://lmstudio.ai).
Recommended quant: `Q4_K_M`.

### Thinking mode

Qwen3.5 supports an optional chain-of-thought `<think>` block before the answer.
In the transformers API this is controlled via `enable_thinking=True/False` in
`apply_chat_template`.

**llama.cpp:** thinking appears to be enabled by default. `--reasoning-budget 0`
and `--chat-template-kwargs '{"enable_thinking":false}'` have been tested but do
not reliably disable it — this may be a llama.cpp issue with Qwen3 thinking models.
Check [llama.cpp issues](https://github.com/ggml-org/llama.cpp/issues) for the
current status.

## Pipeline diagram

![](diagrams/diagram_02.png)

## Acknowledgements

Based on [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B)
by the Qwen Team. If you use this model in research, please cite the original:

```bibtex
@misc{qwen3.5,
    title  = {{Qwen3.5}: Towards Native Multimodal Agents},
    author = {{Qwen Team}},
    month  = {February},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.5}
}
```