ayourtch/Qwen3.8-27B-shoehorn-fits

Qwen3.8-27B — shoehorn budget-exact fits

Four GGUF quantizations of Qwen/Qwen3.8-27B, produced with shoehorn, plus the accuracy benchmark that checks whether they still think.

A preset quant like Q8_0 or IQ4_XS applies one recipe to every tensor, and whatever size falls out is the size you get. shoehorn works the other way round: you give it a VRAM budget, a context length and a KV cache type, and it solves for a per-tensor mix of quant formats that fills whatever is left after inference overhead. The point is to land on a specific card at a specific context, not to hit a particular bits-per-weight number.

So the obvious question is whether solving for a size costs you accuracy. We benchmarked all four against the two stock quants. The short answer is in the table below; the long answer is that at 4.9 bits per weight and up, this benchmark could not measure a difference, and at 2.9 bits it very clearly could.

The files

file	bpw	size	solved for	GiB
qwen38-27b-fit-128k-q4kv.gguf	6.099	20,838,706,944 B	131072 ctx, q4_0 KV, 23.5 GiB budget	19.41
qwen38-27b-fit-192k-q4kv.gguf	5.509	18,825,459,584 B	196608 ctx, q4_0 KV, 23.5 GiB budget	17.53
qwen38-27b-fit-256k-q4kv.gguf	4.920	16,812,197,504 B	262144 ctx, q4_0 KV, 23.5 GiB budget	15.66
qwen38-27b-fit-256k-17g.gguf	2.876	9,832,877,184 B	262144 ctx, q4_0 KV, 17 GiB budget	9.16

Every budget above is a budget for weights plus KV cache plus overhead, at the stated context. That is what "solved for" means here — the 17 GiB file was built to fit a 24 GB card together with its speculative-decoding draft model and a full 256K context, and it does: 21.6 GiB, measured in earlier testing of that configuration.

These are text-only. The base model is multimodal; no vision projector (mmproj) is included here and the vision path was neither converted nor tested.

Exactly which bytes were benchmarked

Every number in this card was produced by these exact files. The published weights are byte-identical to the ones that ran the benchmark — the hashes below were taken on the machine that ran the benchmark, and match what HuggingFace stores for each of the four published files.

file	size (bytes)	sha256
qwen38-27b-fit-128k-q4kv.gguf	20838706944	f30987d23ed027d4ecf5e49aed5a8479162e8d4f796c6e8a541d15e16d08384a
qwen38-27b-fit-192k-q4kv.gguf	18825459584	2faadd35212ae9132e147eb22e423a7d708aed19da1bbc5a5492029806ad99f6
qwen38-27b-fit-256k-q4kv.gguf	16812197504	b156c0c1aa0a2659e4d0ed203f7e6e7f8ff847a081f77f07aebf683cea80eb7a
qwen38-27b-fit-256k-17g.gguf	9832877184	787b2bfafc39177d051b13e5c2359c5950c92496b9ed1fd3f51df750ec78dd6b

Verify with sha256sum <file> after downloading.

The comparison files, which are not in this repo but which the benchmark numbers depend on:

file	size (bytes)	sha256
Qwen3.8-27B-Q8_0.gguf	29047085024	57484e3196aaff8dbbd71c666158b00206baa2bc134e4f16ae90fc8cbdfbeea2
Qwen3.8-27B-IQ4_XS.gguf	15705860064	8b29da0e6cbc5032d478cd2333f5071b5195b4fa903c1d51d86704386c18fb29
Qwen3.8-27B-MTP-ONLY-Q8_0.gguf	4503398688	674d0fc3b2b09c48cf77fbab0aba39b9c4ee538bd240fa87c1f13044260f7d7b

The two stock quants are from unsloth/Qwen3.8-27B-GGUF, and the revision matters. Both files match that repo exactly at any revision before commit 408fcc1807ab (2026-08-14, "Add Unsloth style chat template"). That commit embedded a chat template and grew each file by exactly 1024 bytes, so the current main copies hash differently. If you download them today you will get the +1024-byte versions.

We checked whether that mattered, because it is the kind of difference that quietly breaks a comparison. The fits were built from our own BF16 conversion and carry a different embedded chat template (9993 characters) than the baselines (8945). Different prompt formatting between the two groups would invalidate the whole table.

It does not, for this eval. The harness sends exactly one system message and one user message and uses no tools. Rendering both templates on that input produces byte-identical output — 563 characters, same hash, both resolving reasoning_effort to xhigh. Every difference between the two templates lives in a path this benchmark never takes: multi-system-message merging, tool-call argument handling, and a high→xhigh alias.

Benchmark

92 questions, four sources: AIME2025 (25), GPQA Diamond (25), SuperGPQA (25), COMPSEC (17).

Every leg ran on the same RTX PRO 6000 Blackwell at a 250 W cap, the same llama.cpp build (22b8e31), the same MTP draft model for speculative decoding, q4_0 KV cache, flash attention on, temperature 0, seed 42, 32768 max tokens per answer.

One caveat you will notice in the published CSVs: 24 rows are tagged GPQA Diamond and one is tagged GPQA Diamond (modified). That marker is inherited verbatim from the upstream ds4 corpus — it is antirez's annotation, not a change of ours — and the item is counted within the 25. So a reader tallying source == "GPQA Diamond" gets 24, not 25.

leg	bpw	ctx	accuracy	pass	wrong	exhausted	median case	median completion tok
Qwen3.8-27B-Q8_0 (stock)	8.500	131072	83.7%	77	7	8	51s	3475
Qwen3.8-27B-IQ4_XS (stock)	4.596	131072	84.8%	78	5	9	42s	3721
fit-128k	6.099	131072	88.0%	81	5	6	53s	3612
fit-192k	5.509	196608	88.0%	81	4	7	56s	4178
fit-256k	4.920	262144	87.0%	80	5	7	63s	4436
fit-17g	2.876	131072	67.4%	62	9	21	64s	6464

By source:

leg	AIME2025	COMPSEC	GPQA Diamond	SuperGPQA
Q8_0	84% (21/25)	100% (17/17)	80% (20/25)	76% (19/25)
IQ4_XS	88% (22/25)	100% (17/17)	76% (19/25)	80% (20/25)
fit-128k	96% (24/25)	88% (15/17)	84% (21/25)	84% (21/25)
fit-192k	92% (23/25)	94% (16/17)	84% (21/25)	84% (21/25)
fit-256k	92% (23/25)	100% (17/17)	80% (20/25)	80% (20/25)
fit-17g	64% (16/25)	100% (17/17)	48% (12/25)	68% (17/25)

Note that the three 23.5 GiB fits each ran at the context they were solved for, while the stock quants and fit-17g ran at 131072. That is deliberate — running a fit at its target context is the thing the fit exists to do.

The comparison worth looking at twice is fit-256k against stock IQ4_XS: 4.920 bpw against 4.596, 15.66 GiB against 14.63, and 87.0% against 84.8% — but fit-256k carried a 262144-token context while IQ4_XS ran at 131072.

Read the top five rows as one cluster

The fits come out a few points above Q8_0. That is not a result. The comparison is paired, so only questions where two legs disagree carry information, and there are very few of them: fit-128k versus Q8_0 is 8 discordant questions out of 92, splitting 6–2 in the fit's favour, McNemar exact p = 0.29. Every pairwise comparison among those five legs is non-significant.

Simulating the exact test at the discordance rate these legs actually produce, the power at 92 questions is 0.17. Eighty percent power would need about 400 questions. Even a perfect effect — every discordant question falling the same way — only reaches 0.82 at this sample size.

So the defensible claim is no measurable accuracy loss against Q8_0 down to 4.920 bpw, and explicitly not that the fits are better than Q8_0. If you want a ranking among those five, this benchmark cannot give you one.

What happens at 2.876 bpw

The 2.876 bpw file is the one comparison here that clears the bar. Against Q8_0: 19-vs-4 discordant, p = 0.003, power 0.88. Against fit-256k: 20-vs-2, p = 0.0001.

But how it degrades is more useful than the headline. Its answered-only accuracy is 87.3% (62/71) against Q8_0's 91.7% (77/84) — a gap of about 4 points, not 16. The headline gap is mostly exhaustion: 21 cases ran out of the 32768-token budget before committing to an answer, against 6–9 for every other leg, and its median completion length is 6464 tokens against 3475–4436.

It is not mostly answering wrongly, and it is not uniformly slower to think either. On the questions it finishes, its median completion is 3818 tokens against Q8_0's 2956 — about 29% longer, not double. The doubled headline median is the upper tail: its third quartile is 28824 tokens, against 11k–13.5k for every other leg (exclusive quartiles). Most questions look normal; a subset runs away and hits the wall.

The collapse is concentrated where the reasoning is longest: GPQA Diamond falls to 48% while COMPSEC stays at 100%.

A larger token budget would likely recover part of that 16 points. We have not tested that, and until someone does, treat 67.4% as "this configuration, at 32768 tokens" rather than as the model's ceiling.

The same questions fail in every leg

Across all six legs, 58 exhaustion events land on only 22 distinct questions, and 4 questions exhaust in all six legs — including Q8_0. Under independence with the same per-leg counts you would expect 0.000 questions to exhaust everywhere, and a permutation test over question ids gives p < 5e-6 (200,000 trials).

Some questions are simply long-reasoning traps, and that property survives across quant families and across a 3x range in bits per weight.

Running them

llama-server \
  -m qwen38-27b-fit-128k-q4kv.gguf \
  -md Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 \
  -c 131072 -ctk q4_0 -ctv q4_0 -fa on \
  -ngl 99 -ngld 99 --host 127.0.0.1 --port 8083

Set -c to the context the file was solved for. The KV type is not optional: these were solved against q4_0 KV, and running f16 KV changes the memory arithmetic the fit was built around.

-ngld 99 matters — without it the draft model runs on the CPU and speculative decoding ends up slower than not doing it at all.

The draft model used here is an MTP-only Q8_0 extraction from the same base. It is not included in this repo.

How the fits were made

shoehorn --ctx 131072 --kv q4_0 --budget 23.5GiB --calibrate
shoehorn --ctx 196608 --kv q4_0 --budget 23.5GiB --calibrate
shoehorn --ctx 262144 --kv q4_0 --budget 23.5GiB --calibrate --force-calibrate
shoehorn --ctx 262144 --kv q4_0 --budget 17GiB   --calibrate --force-calibrate

Starting from a BF16 GGUF and an imatrix generated with llama-imatrix on bartowski's calibration text.

--calibrate is the part that does the work. It writes the file, launches llama.cpp once, reads back the measured allocations and re-solves against them. At 128K the KV estimate was 9.14 GiB and the real allocation was 2.25 GiB — most layers in a hybrid model don't keep a full-attention cache — and the re-solve handed 5.75 GiB back to the weights.

--force-calibrate is our own patch, on a fork. Three estimate-based gates bail out before --calibrate ever runs, which means the tool can refuse a job on the strength of an estimate it is about to discover is wrong. The flag turns those gates into warnings. Both 256K fits need it.

On a multi-GPU box, pass --budget explicitly — the NVML probe reads device 0 only.

What's in this repo

Per-leg results for all six benchmark legs live under bench/: the summary JSON, the per-case CSV (question id, source, domain, pass/fail, token counts, timings) and a config-public.txt recording the exact server command, sampler settings, build and driver version for that leg.

The evaluation harness itself is ours and is not public, so the eval is not reproducible from this repo — the inference side is (the server command is recorded per leg, and the question sources are named above).

The raw model outputs are not published. They quote the benchmark questions verbatim, and 25 of the 92 come from GPQA Diamond, whose authors ask that examples not be reproduced online. The per-case CSVs carry question ids and results but no question text.

Credits and licences

• Base model: Qwen/Qwen3.8-27B by the Qwen team, Apache-2.0. These are derived weights and the base model's licence terms govern them; the quantization does not change what you may do with the model.
• Quantizer: shoehorn, MIT. The budget solver, the calibration loop and the per-tensor format assignment are all its work. The --force-calibrate flag is a local patch on a fork, not upstream.
• Evaluation: antirez/ds4 ("DwarfStar") by Salvatore Sanfilippo, MIT — an inference engine for DeepSeek V4 Flash and GLM 5.2 on consumer hardware. Our harness is a Rust port of its ds4_bench.c and ds4_eval.c, and the 92-question selection is antirez's. These numbers would not exist without it.
• Inference: llama.cpp, MIT.

Benchmark questions originate with their respective authors: AIME2025, GPQA Diamond, SuperGPQA, and a computer-security set. Because the question set is public, treat these numbers as a relative comparison between quantizations run under identical conditions — not as leaderboard-comparable scores.