---
base_model: llmfan46/Qwen3.5-27B-heretic
quantized_by: alexdenton
tags:
  - gguf
  - qwen3.5
  - llama-cpp
  - quantized
library_name: gguf
license: apache-2.0
---

# Qwen3.5-27B-heretic GGUF

GGUF quantizations of [llmfan46/Qwen3.5-27B-heretic](https://huggingface.co/llmfan46/Qwen3.5-27B-heretic).

## Available quantizations

| File | Quant | Size | BPW | Quality |
|------|-------|------|-----|---------|
| `Qwen3.5-27B-heretic-Q8_0.gguf` | Q8_0 | 27 GB | 8.50 | Excellent — near-lossless |
| `Qwen3.5-27B-heretic-Q6_K.gguf` | Q6_K | 21 GB | 6.57 | Very good |
| `Qwen3.5-27B-heretic-Q5_K_M.gguf` | Q5_K_M | 18 GB | 5.77 | Good |
| `Qwen3.5-27B-heretic-Q4_K_M.gguf` | Q4_K_M | 15.4 GB | 4.92 | Recommended for most users |
| `Qwen3.5-27B-heretic-Q3_K_M.gguf` | Q3_K_M | 12.4 GB | 3.95 | Usable — for low VRAM setups |

## Multimodal projector (vision)

| File | Format | Size |
|------|--------|------|
| `Qwen3.5-27B-mmproj-BF16.gguf` | BF16 | 889 MB |

## How to use

### llama-server (recommended for API usage)

```bash
llama-server \
    -m Qwen3.5-27B-heretic-Q8_0.gguf \
    --mmproj Qwen3.5-27B-mmproj-BF16.gguf \
    --host 0.0.0.0 \
    --ctx-size 32768 \
    --gpu-layers 99 \
    --tensor-split 1,1 \
    --chat-template-kwargs '{"enable_thinking": false}' \
    --repeat-penalty 1.0 \
    --presence-penalty 1.5 \
    --min-p 0.0 \
    --top-k 20 \
    --top-p 0.8 \
    --temp 0.7
```

### llama-cli

```bash
llama-cli -m Qwen3.5-27B-heretic-Q4_K_M.gguf -ngl 99 -c 4096
```

## Which quantization should I use?

- **Q8_0**: Best quality, needs ~28 GB VRAM. Fits on 2x RTX 3090.
- **Q6_K**: Great quality, needs ~22 GB VRAM. Fits on 1x RTX 3090.
- **Q5_K_M**: Good balance of quality and size. Fits on 1x RTX 3090 with room for context.
- **Q4_K_M**: Most popular choice. Fits on 1x RTX 4070 Ti Super / RTX 3090.
- **Q3_K_M**: For constrained VRAM setups (~13 GB needed).

## Quantization details

- Quantized with [llama.cpp](https://github.com/ggml-org/llama.cpp) `convert_hf_to_gguf.py` + `llama-quantize`
- Original model: [llmfan46/Qwen3.5-27B-heretic](https://huggingface.co/llmfan46/Qwen3.5-27B-heretic)