Empero AI
# Qwythos-9B-v2-GGUF GGUF quantizations of **[empero-ai/Qwythos-9B-v2](https://huggingface.co/empero-ai/Qwythos-9B-v2)** for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes. **Qwythos-9B-v2 is the new and improved Qwythos** — all the deep chain-of-thought reasoning of the base Qwythos, with the **looping behavior fixed**. The looping/degeneration that showed up under greedy or low-temperature decoding is trained out (**6.7% → 0%**), the native **MTP head is restored**, and the identity prompt is cleaned up — while knowledge and reasoning are held at (or above) the base Qwythos level. The fix uses **FTPO (Final-Token Preference Optimization)**: the exact token that *starts* a repetition loop is identified and the model is gently trained to prefer coherent alternatives at that one position, leaving the rest of the distribution — and therefore its knowledge and reasoning — untouched. For full training details, evaluation numbers, and sample generations, see the **[base model card](https://huggingface.co/empero-ai/Qwythos-9B-v2)**. --- ## What's new vs. the base Qwythos - 🔁 **Looping behavior eliminated** — repetition under greedy / low-temp decoding dropped **6.7% → 0%**. **Greedy decoding is now safe** — you no longer need `repeat-penalty` as a band-aid. - 🧩 **MTP head restored** — the native multi-token-prediction module is back in the `-MTP-` files, so speculative-decoding (`--spec-type draft-mtp`) works. - 🧠 **Reasoning preserved** — MMLU / GSM8K / GPQA / ARC held at or above the base Qwythos level (see the model card). - 🪪 **Cleaner identity** — states who it is once, only when asked. - 🔓 Still intentionally **uncensored**, still **1M-token context** (YaRN), still **multimodal-capable** (Qwen3.5 vision tower). --- ## Files ### Normal text weights — trunk only (32 blocks) | File | Quant | Size | Notes | |---|---|---|---| | `Qwythos-9B-v2-Q4_K_M.gguf` | Q4_K_M | 5.34 GiB / 5.74 GB | **recommended default** — smallest practical, good quality | | `Qwythos-9B-v2-Q5_K_M.gguf` | Q5_K_M | 6.08 GiB / 6.52 GB | balanced quality / size | | `Qwythos-9B-v2-Q6_K.gguf` | Q6_K | 6.95 GiB / 7.46 GB | high quality | | `Qwythos-9B-v2-Q8_0.gguf` | Q8_0 | 8.87 GiB / 9.53 GB | near-lossless | | `Qwythos-9B-v2-BF16.gguf` | BF16 | 16.69 GiB / 17.92 GB | full precision (conversion base) | If you don't know which to pick, **Q4_K_M is the right starting point.** ### MTP-enabled text weights (33 blocks, `nextn_predict_layers = 1`) These embed the restored Qwen3.5-compatible MTP head. Use them with llama.cpp builds that support MTP draft speculation (`--spec-type draft-mtp`). The MTP matrices are retained at **Q8_0** in every quantized variant. | File | Quant | Size | Notes | |---|---|---|---| | `Qwythos-9B-v2-MTP-Q4_K_M.gguf` | Q4_K_M + MTP | 5.50 GiB / 5.90 GB | **recommended MTP default** | | `Qwythos-9B-v2-MTP-Q5_K_M.gguf` | Q5_K_M + MTP | 6.25 GiB / 6.71 GB | balanced quality / size | | `Qwythos-9B-v2-MTP-Q6_K.gguf` | Q6_K + MTP | 7.14 GiB / 7.67 GB | high quality | | `Qwythos-9B-v2-MTP-Q8_0.gguf` | Q8_0 + MTP | 9.11 GiB / 9.79 GB | near-lossless | | `Qwythos-9B-v2-MTP-BF16.gguf` | BF16 + MTP | 17.14 GiB / 18.41 GB | full precision (conversion base) | ### Vision projector — for image input | File | Size | Notes | |---|---|---| | `mmproj-Qwythos-9B-v2-BF16.gguf` | 0.86 GiB / 0.92 GB | CLIP-style vision encoder + projector at **BF16** native precision; **required for images**, pairs with any text quant above | The vision tower is inherited **unchanged from Qwen3.5-9B** — it was frozen through both the base Qwythos SFT and the v2 FTPO fine-tune, so image behavior matches base Qwen3.5-9B. This mmproj is interchangeable with any Qwen3.5-9B `mmproj-*.gguf`. --- ## Hybrid-precision quantization (Gated-DeltaNet / SSM tensors) Qwythos is a **hybrid** model — a 3:1 mix of Gated-DeltaNet linear-attention (SSM) blocks and full-attention blocks. The linear-attention state tensors are disproportionately sensitive to low-bit quantization, so the K-quants here keep them at higher precision than the surrounding weights: | Quant | `ssm_alpha` | `ssm_beta` | `ssm_out` | |---|---|---|---| | **Q6_K** | Q8_0 | Q8_0 | Q8_0 | | **Q5_K_M** | Q8_0 | Q8_0 | Q6_K | | **Q4_K_M** | Q8_0 | Q8_0 | Q6_K | The remaining SSM state tensors (`ssm_a`, `ssm_conv1d`, `ssm_dt`, `ssm_norm`) are kept at **F32** by the converter. This preserves the hybrid/SSM blocks for a small (~2–4%) increase in file size over a flat K-quant. `Q8_0` and `BF16` are uniform and need no overrides. --- ## Quick start ### llama.cpp ```bash llama-cli \ -m Qwythos-9B-v2-Q4_K_M.gguf \ -p "Walk through the biochemistry of how organophosphate nerve agents inhibit acetylcholinesterase." \ -n 8192 \ --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \ -c 16384 ``` Because v2's looping is trained out, `--repeat-penalty` is now optional and **greedy decoding (`--temp 0`) stays coherent.** ### Ollama ```bash ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M ``` ### LM Studio / jan / KoboldCpp Drop any `.gguf` into your runtime's model directory. Qwythos uses the standard Qwen3.5 chat template; modern GGUF runtimes load it automatically from the file. ### MTP draft speculation ```bash llama-server \ -m Qwythos-9B-v2-MTP-Q4_K_M.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 6 \ -c 16384 --port 8080 ``` MTP support requires a recent llama.cpp build. If your runtime doesn't support MTP yet, use the normal files. --- ## Vision (image input) Download a text quant **plus** the `mmproj-*.gguf`, then run llama.cpp's multimodal CLI/server: ```bash llama-mtmd-cli \ -m Qwythos-9B-v2-Q4_K_M.gguf \ --mmproj mmproj-Qwythos-9B-v2-BF16.gguf \ --image ./photo.jpg \ -p "Describe this image in detail." \ --temp 0.6 --top-p 0.95 --top-k 20 -c 16384 ``` **Honest note:** all Qwythos training (base SFT and v2 FTPO) was **text-only** — the vision tower was never fine-tuned, so image-grounded reasoning inherits base Qwen3.5-9B behavior and has not been independently evaluated for this release. --- ## Sampling recommendations Qwythos is a reasoning model — every response opens with a `