Instructions to use empero-ai/Qwythos-9B-v2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use empero-ai/Qwythos-9B-v2-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="empero-ai/Qwythos-9B-v2-GGUF",
	filename="Qwythos-9B-v2-BF16.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use empero-ai/Qwythos-9B-v2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Use Docker

docker model run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use empero-ai/Qwythos-9B-v2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "empero-ai/Qwythos-9B-v2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "empero-ai/Qwythos-9B-v2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Ollama
How to use empero-ai/Qwythos-9B-v2-GGUF with Ollama:
```
ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
```

Unsloth Studio

How to use empero-ai/Qwythos-9B-v2-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for empero-ai/Qwythos-9B-v2-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for empero-ai/Qwythos-9B-v2-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for empero-ai/Qwythos-9B-v2-GGUF to start chatting

How to use empero-ai/Qwythos-9B-v2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use empero-ai/Qwythos-9B-v2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use empero-ai/Qwythos-9B-v2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use empero-ai/Qwythos-9B-v2-GGUF with Docker Model Runner:
```
docker model run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
```

Lemonade

How to use empero-ai/Qwythos-9B-v2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwythos-9B-v2-GGUF-Q4_K_M

List all available models

lemonade list

Qwythos-9B-v2-GGUF / README.md

empero-ai

Update README.md

97c11b0 verified 5 days ago

preview code

Raw

History Blame Contribute Delete

8.65 kB

	---
	license: apache-2.0
	base_model: empero-ai/Qwythos-9B-v2
	base_model_relation: quantized
	language:
	- en
	pipeline_tag: image-text-to-text
	library_name: gguf
	tags:
	- gguf
	- llama.cpp
	- quantized
	- qwythos
	- qwen3.5
	- ftpo
	- reasoning
	- uncensored
	- long-context
	- 1M-context
	- function-calling
	- multimodal
	- vision
	---

	<p align="center">
	<img src="https://huggingface.co/empero-ai/Qwythos-9B-v2/resolve/main/qwythos_v2.png" alt="Qwythos-9B-v2" width="640"/>
	</p>

	<p align="center"><b>Empero AI</b></p>

	# Qwythos-9B-v2-GGUF

	GGUF quantizations of [empero-ai/Qwythos-9B-v2](https://huggingface.co/empero-ai/Qwythos-9B-v2) for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes.

	Qwythos-9B-v2 is the new and improved Qwythos — all the deep chain-of-thought reasoning of the base Qwythos, with the looping behavior fixed. The looping/degeneration that showed up under greedy or low-temperature decoding is trained out (6.7% → 0%), the native MTP head is restored, and the identity prompt is cleaned up — while knowledge and reasoning are held at (or above) the base Qwythos level.

	The fix uses FTPO (Final-Token Preference Optimization): the exact token that starts a repetition loop is identified and the model is gently trained to prefer coherent alternatives at that one position, leaving the rest of the distribution — and therefore its knowledge and reasoning — untouched.

	For full training details, evaluation numbers, and sample generations, see the [base model card](https://huggingface.co/empero-ai/Qwythos-9B-v2).

	---

	## What's new vs. the base Qwythos

	- 🔁 Looping behavior eliminated — repetition under greedy / low-temp decoding dropped 6.7% → 0%. Greedy decoding is now safe — you no longer need `repeat-penalty` as a band-aid.
	- 🧩 MTP head restored — the native multi-token-prediction module is back in the `-MTP-` files, so speculative-decoding (`--spec-type draft-mtp`) works.
	- 🧠 Reasoning preserved — MMLU / GSM8K / GPQA / ARC held at or above the base Qwythos level (see the model card).
	- 🪪 Cleaner identity — states who it is once, only when asked.
	- 🔓 Still intentionally uncensored, still 1M-token context (YaRN), still multimodal-capable (Qwen3.5 vision tower).

	---

	## Files

	### Normal text weights — trunk only (32 blocks)

	\| File \| Quant \| Size \| Notes \|
	\|---\|---\|---\|---\|
	\| `Qwythos-9B-v2-Q4_K_M.gguf` \| Q4_K_M \| 5.34 GiB / 5.74 GB \| recommended default — smallest practical, good quality \|
	\| `Qwythos-9B-v2-Q5_K_M.gguf` \| Q5_K_M \| 6.08 GiB / 6.52 GB \| balanced quality / size \|
	\| `Qwythos-9B-v2-Q6_K.gguf` \| Q6_K \| 6.95 GiB / 7.46 GB \| high quality \|
	\| `Qwythos-9B-v2-Q8_0.gguf` \| Q8_0 \| 8.87 GiB / 9.53 GB \| near-lossless \|
	\| `Qwythos-9B-v2-BF16.gguf` \| BF16 \| 16.69 GiB / 17.92 GB \| full precision (conversion base) \|

	If you don't know which to pick, Q4_K_M is the right starting point.

	### MTP-enabled text weights (33 blocks, `nextn_predict_layers = 1`)

	These embed the restored Qwen3.5-compatible MTP head. Use them with llama.cpp builds that support MTP draft speculation (`--spec-type draft-mtp`). The MTP matrices are retained at Q8_0 in every quantized variant.

	\| File \| Quant \| Size \| Notes \|
	\|---\|---\|---\|---\|
	\| `Qwythos-9B-v2-MTP-Q4_K_M.gguf` \| Q4_K_M + MTP \| 5.50 GiB / 5.90 GB \| recommended MTP default \|
	\| `Qwythos-9B-v2-MTP-Q5_K_M.gguf` \| Q5_K_M + MTP \| 6.25 GiB / 6.71 GB \| balanced quality / size \|
	\| `Qwythos-9B-v2-MTP-Q6_K.gguf` \| Q6_K + MTP \| 7.14 GiB / 7.67 GB \| high quality \|
	\| `Qwythos-9B-v2-MTP-Q8_0.gguf` \| Q8_0 + MTP \| 9.11 GiB / 9.79 GB \| near-lossless \|
	\| `Qwythos-9B-v2-MTP-BF16.gguf` \| BF16 + MTP \| 17.14 GiB / 18.41 GB \| full precision (conversion base) \|

	### Vision projector — for image input

	\| File \| Size \| Notes \|
	\|---\|---\|---\|
	\| `mmproj-Qwythos-9B-v2-BF16.gguf` \| 0.86 GiB / 0.92 GB \| CLIP-style vision encoder + projector at BF16 native precision; required for images, pairs with any text quant above \|

	The vision tower is inherited unchanged from Qwen3.5-9B — it was frozen through both the base Qwythos SFT and the v2 FTPO fine-tune, so image behavior matches base Qwen3.5-9B. This mmproj is interchangeable with any Qwen3.5-9B `mmproj-*.gguf`.

	---

	## Hybrid-precision quantization (Gated-DeltaNet / SSM tensors)

	Qwythos is a hybrid model — a 3:1 mix of Gated-DeltaNet linear-attention (SSM) blocks and full-attention blocks. The linear-attention state tensors are disproportionately sensitive to low-bit quantization, so the K-quants here keep them at higher precision than the surrounding weights:

	\| Quant \| `ssm_alpha` \| `ssm_beta` \| `ssm_out` \|
	\|---\|---\|---\|---\|
	\| Q6_K \| Q8_0 \| Q8_0 \| Q8_0 \|
	\| Q5_K_M \| Q8_0 \| Q8_0 \| Q6_K \|
	\| Q4_K_M \| Q8_0 \| Q8_0 \| Q6_K \|

	The remaining SSM state tensors (`ssm_a`, `ssm_conv1d`, `ssm_dt`, `ssm_norm`) are kept at F32 by the converter. This preserves the hybrid/SSM blocks for a small (~2–4%) increase in file size over a flat K-quant. `Q8_0` and `BF16` are uniform and need no overrides.

	---

	## Quick start

	### llama.cpp

	```bash
	llama-cli \
	-m Qwythos-9B-v2-Q4_K_M.gguf \
	-p "Walk through the biochemistry of how organophosphate nerve agents inhibit acetylcholinesterase." \
	-n 8192 \
	--temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
	-c 16384
	```

	Because v2's looping is trained out, `--repeat-penalty` is now optional and greedy decoding (`--temp 0`) stays coherent.

	### Ollama

	```bash
	ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
	```

	### LM Studio / jan / KoboldCpp

	Drop any `.gguf` into your runtime's model directory. Qwythos uses the standard Qwen3.5 chat template; modern GGUF runtimes load it automatically from the file.

	### MTP draft speculation

	```bash
	llama-server \
	-m Qwythos-9B-v2-MTP-Q4_K_M.gguf \
	--spec-type draft-mtp \
	--spec-draft-n-max 6 \
	-c 16384 --port 8080
	```

	MTP support requires a recent llama.cpp build. If your runtime doesn't support MTP yet, use the normal files.

	---

	## Vision (image input)

	Download a text quant plus the `mmproj-*.gguf`, then run llama.cpp's multimodal CLI/server:

	```bash
	llama-mtmd-cli \
	-m Qwythos-9B-v2-Q4_K_M.gguf \
	--mmproj mmproj-Qwythos-9B-v2-BF16.gguf \
	--image ./photo.jpg \
	-p "Describe this image in detail." \
	--temp 0.6 --top-p 0.95 --top-k 20 -c 16384
	```

	Honest note: all Qwythos training (base SFT and v2 FTPO) was text-only — the vision tower was never fine-tuned, so image-grounded reasoning inherits base Qwen3.5-9B behavior and has not been independently evaluated for this release.

	---

	## Sampling recommendations

	Qwythos is a reasoning model — every response opens with a `<think>...</think>` block before the answer.

	\| Parameter \| Value \|
	\|---\|---\|
	\| `temperature` \| 0.6 \|
	\| `top_p` \| 0.95 \|
	\| `top_k` \| 20 \|
	\| `repeat_penalty` \| 1.05 (optional in v2) \|
	\| `max_new_tokens` \| 16384 \|

	Unlike the base Qwythos, v2 does not loop under greedy / low-temperature decoding — you can use `--temp 0` for deterministic runs without repetition. The 0.6-temperature settings above still match Qwen3.5's official thinking-mode recommendations for best quality.

	---

	## Long context (1M tokens)

	The GGUFs ship with YaRN rope-scaling baked in for a 1,048,576-token context window (4× the 262,144 native). Set `-c` up to `1048576`; lower it to reduce KV-cache memory for shorter prompts. A single H100/H200-class GPU comfortably handles 256k–512k; the full 1M typically needs multi-GPU or aggressive KV-cache offload.

	---

	## Conversion & verification

	- Converted and quantized with llama.cpp (`convert_hf_to_gguf.py`, `llama-quantize`), architecture `qwen35`, GGUF v3.
	- MTP variants: default conversion (33-block, `nextn_predict_layers = 1`, 15 MTP tensors, MTP matrices pinned Q8_0). Normal variants: `--no-mtp` (32-block trunk-only). mmproj: `--mmproj --outtype bf16`.
	- Hybrid-precision overrides applied per the table above.
	- Structurally verified (arch / block count / `nextn` key / per-tensor types) and smoke-tested for load + coherent generation.
	- `shasum -a 256 -c SHA256SUMS` covers all 11 artifacts.

	---

	## License & acknowledgements

	Apache-2.0, inherited from Qwen3.5-9B. Shared for research and experimentation, as-is.

	- Developed and released by Empero AI
	- Base model: Qwen3.5-9B (Alibaba Qwen team)
	- Looping fixed with FTPO (Final-Token Preference Optimization)
	- Quantization: llama.cpp (ggml-org)
	- HF model: [empero-ai/Qwythos-9B-v2](https://huggingface.co/empero-ai/Qwythos-9B-v2)