---
language:
  - en
  - ru
license: apache-2.0
library_name: gguf
base_model:
  - WaveCut/Qwythos-9B-v2-Heretic
base_model_relation: quantized
tags:
  - heretic
  - uncensored
  - abliteration
  - gguf
  - llama.cpp
  - qwen3.5
pipeline_tag: text-generation
---

# Qwythos-9B-v2-Heretic-GGUF

**GGUF quantizations** of [`WaveCut/Qwythos-9B-v2-Heretic`](https://huggingface.co/WaveCut/Qwythos-9B-v2-Heretic) — the Heretic-decensored version of [`empero-ai/Qwythos-9B-v2`](https://huggingface.co/empero-ai/Qwythos-9B-v2). Use these with [llama.cpp](https://github.com/ggml-org/llama.cpp), [Ollama](https://ollama.com), [LM Studio](https://lmstudio.ai), [KoboldCpp](https://github.com/LostRuins/koboldcpp), or any other GGUF runtime.

## Files

| File | Format | ~Bits/weight | Size | Recommended use |
|---|---|---|---|---|
| `Qwythos-9B-v2-Heretic.Q4_K_M.gguf` | K-quant (mixed) | 4.5 | ~5.6 GB | **Best 4-bit default** — fits in ~6.5 GB VRAM, balanced quality/size |
| `Qwythos-9B-v2-Heretic.Q5_K_M.gguf` | K-quant (mixed) | 5.5 | ~6.5 GB | Higher quality 5-bit, ~7.5 GB VRAM |
| `Qwythos-9B-v2-Heretic.Q6_K.gguf` | K-quant (mixed) | 6.6 | ~7.4 GB | Very close to FP16, ~8.5 GB VRAM |
| `Qwythos-9B-v2-Heretic.Q8_0.gguf` | 8-bit symmetric | 8.5 | ~9.5 GB | Effectively lossless, ~10.5 GB VRAM |

## Quantization recipe

| Step | Tool | Version |
|---|---|---|
| HF → F16 GGUF | `convert_hf_to_gguf.py` | llama.cpp `b9986` (commit `91c631b`), run with **`--no-mtp`** |
| Quantize | `llama-quantize` | llama.cpp `b9986` (prebuilt linux-x64) |

**Note on `--no-mtp`:** the base Qwen3.5 model bundles a multi-token-prediction (MTP) head as block #32. Excluding it via `--no-mtp` produces a clean 32-block text GGUF that loads correctly in current llama.cpp builds. If you want speculative decoding with the MTP head, run `convert_hf_to_gguf.py --mtp` separately.

No `imatrix` (importance matrix) calibration was used — K-quants hold up well without one for this model size, and Q8_0 is format-defined and never benefits from it.

## Architecture

Qwen3.5 **hybrid** — 32 blocks mixing attention and SSM (Mamba-style) layers. Supported in llama.cpp `b9986` and later under the `qwen35` architecture key.

## Usage

```bash
# llama.cpp
llama-cli -m Qwythos-9B-v2-Heretic.Q4_K_M.gguf -p "Hello" --chat-template chat_template.jinja

# Ollama (create a Modelfile pointing to the .gguf)
ollama create qwythos-heretic -f Modelfile
ollama run qwythos-heretic
```

## Disclaimer

Uncensored (safety alignment removed via Heretic). The original `empero-ai/Qwythos-9B-v2` maintainers are not affiliated with this derivative. Use responsibly.