steampunque/Qwen3.5-9B-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3.5-9B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.5-9B

The mixed precision quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 8/27/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOSD"],
   [0 ,"Q5_K_6658"], [1 ,"Q5_K_6655"], [2 ,"Q4_K_6655"], [3 ,"Q4_K_6555"],
   [4 ,"Q4_K_5555"], [5 ,"Q4_K_5545"], [6 ,"Q4_K_5545"], [7 ,"Q4_K_6555"],
   [8 ,"Q4_K_5444"], [9 ,"Q4_K_5444"], [10,"Q4_K_5444"], [11,"Q4_K_6554"],
   [12,"Q4_K_5444"], [13,"Q4_K_5444"], [14,"Q4_K_5444"], [15,"Q4_K_6554"],
   [16,"Q4_K_5554"], [17,"Q4_K_5554"], [18,"Q4_K_5554"], [19,"Q4_K_6555"],
   [20,"Q4_K_5555"], [21,"Q4_K_5555"], [22,"Q4_K_5555"], [23,"Q4_K_6555"],
   [24,"Q4_K_5565"], [25,"Q4_K_5565"], [26,"Q4_K_5565"], [27,"Q5_K_6665"],
   [28,"Q5_K_6666"], [29,"Q5_K_6666"], [30,"Q5_K_6668"], [31,"Q6_K_8666"],
   [32,"Q4_K_6554"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized for very strong performance across a set of curated reasoning prompts. The final quant size is about 0.4B bigger than Q4_K_M. The quant includes a 0.2B MTP layer supported by llama.cpp b9180 and above (versions less than b9180 will not load this quant)

Comparison:

Quant	size	PPL	Comment
Q4_K_M	5.6e9	7.7	Q4_K_M with default embedding and output
Q4_E_H	6.2e9	7.8	Mixed precision quant with Q6_K embedding Q6_K and ~0.2B MTP layer

Usage:

Qwen3.5-9B is a vision capable dense RL model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Straightforward speculation does not work with the model due to the attention scheme it uses. As of llama.cpp b9180 MTP support for the model was added to upstream but has not been tested.

On a 4070 with all layers and context in VRAM with no vision tower and checkpoints disabled approx performance is:

Q	QKV	NKV	gen tps
Q4_E_H	F16	190k+	72
Q4_E_H	Q8_0	300k+	73

High context yarn config is as follows: set base context for yarn rope scale compute to 262144 (256k), then with a context of 300k tokens the rope scale = 300 / 256 = 1.17.

Then on model start pass --rope-scaling yarn --yarn-orig-ctx 262144 --rope_scale 1.17 (must be ajusted if kv other than 300k)

Later versions of llama.cpp have a bug which soft caps context length to the training context, effectively disabling yarn context extension. Patch server-context.cpp according to https://github.com/ggml-org/llama.cpp/issues/22140 to fix it.

The model appears to be trained to decide itself whether to do a think block or not. When it does a think block it can falls into very heavy overthinking on some prompts and sometimes gets stuck in rep loops with greedy sampling. Over a small set of eval reasoning prompts the model did extremely well, scoring essentially 100% across the eval set. To avoid the overthinking inject think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into a think block inject a bootstrap think start following the assistant prompt:

"<think>\n"

The model was found to be highly capable on reasoning tasks when skipping think block. The model can fall into infinite rep loops on tricky/ambigous prompts when using greedy sampling. This is similar behaviour to other qwen3 thinkers which have trouble with infinite repeat when using greedy sampling particularly at smaller quant sizes (<10B params)

VISION:

The model was tested in vision mode on a couple pretty tough bird ID image and extremely well, with concise and accurate think block accurate final conclusion.

CODE:

The model was tested across a small set of code gen prompts and found to be quite intermittent in its ability to generate working code, and often falls into infinite repeat on the code prompts where it decided to use a think block when using greedy sampling. The model is capable of generating working programs on some prompts.

LONG CONTEXT:

Long context test (needle in haystack) was tested and passed with fast prompt processing ranging from 3000tps at start of 85k prompt to ~2000 tps at end of prompt, making large context actually usable with the model.

The quant was tested against https://huggingface.co/datasets/steampunque/benchlm/blob/main/Qwen3_Runescape_Massive_Prompt.txt and solved the prompt correctly using greedy deterministic sampling with think block forced (it gives wrong answer if think block is not forced)

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3.5-9B.Q4_E_H.gguf	Q4_E_H	6.2e9 B	~0.4B bigger than Q4_K_M, includes ~0.2B MTP layer
Qwen3.5-9B.mmproj.gguf	F16	0.92e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040