ISTA-DASLab

/

Qwen3.8-27B-GSQ-RCO-GGUF

Like 829

Follow

IST Austria Distributed Algorithms and Systems Lab 707

Thank you very much for bringing this issue to our attention.

We are planning to release some standard K-Quant variants as well. Since we will probably release only one or two of them initially, we would be very interested to hear from the community: which exact bitwidths would be the most useful to you?

Regarding multilingual performance, our calibration data intentionally contains multilingual samples, specifically to help preserve the multilingual capabilities of the original model during quantization. We would be especially happy to hear about real-world results across different languages. If you test the models in Persian or any other non-English language, please share your observations and comparisons with us. These insights would be extremely valuable for understanding how well the quantized models preserve multilingual performance in practice.

Thank you so much for the quick and positive response! It is fantastic to hear that you are planning to release K-Quant variants and that your calibration data explicitly includes multilingual samples.

Based on my extensive testing with the base model on an AMD RX 6800 (16GB VRAM) + Lm Studio v0.4.21 with Rocm llama.cpp 2.22.0 gguf engine, I can provide some precise real-world metrics to help guide your upcoming releases.

My Baseline Benchmark (Base Model with Uniform Q3_K_XL):

• Configuration: Q3_K_XL (12.2 GB) + Vision Projector (930 MB) + Q4_0 Flash Attention / KV Cache Quantization + 32,768 Context Length.
• Performance: I achieve an excellent generation speed of ~30 tokens/second, which holds up remarkably well even when the context window is near its maximum capacity.
• Multilingual (Persian/Farsi) Observations: The baseline model’s Persian comprehension is roughly 90% accurate. While it understands prompts perfectly, it occasionally outputs minor character encoding/token rendering glitches in Farsi script. Since my primary use case involves generating English code from Persian prompts, minor output text glitches are perfectly acceptable as long as the prompt comprehension remains flawless.

Suggested Bitwidths and Optimizations for the Community:

Given that 16GB GPUs represent a massive portion of local LLM users, the following two formats optimized with your GSQ-RCO methodology would be absolute game-changers:

1. Q3_K_XL (or Q3_K_L): This is the ultimate sweet spot for speed and memory efficiency. Combined with MTP (Mixed-Tensor Precision) to keep the file size as minimal as possible, it ensures the model never overflows out of the 16GB VRAM, leaving plenty of headroom for large KV Caches (like 32k context) and system usage.
2. Q4_K_M: This would be perfect for users aiming for maximum reasoning accuracy, pushing right up to the 16GB VRAM limit for short-to-medium conversations.

Thank you again for taking this feedback into consideration and for your amazing contribution to open-source LLM research!

Hm, i am on amd igpu, vulkan llama cpp, and it seems to run at full speed as any other quant i usually run. So i haven't noticed any slowdown, task manager shows 100% compute usage. Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf.
I had this problem before with iq quants, when igpu usage was only at 80%, but not this time.

Adding a datapoint on the AMD i-quant prefill question, since I've been benchmarking GSQ-RCO IQ3_S-mtp against Unsloth Dynamic on RDNA3.

Hardware: RX 7800 XT (gfx1101), 16GB, Windows, llama.cpp ROCm/HIP build. All runs --parallel 1 --fit off --n-gpu-layers 66 -fa on -ctk q4_0 -ctv q4_0, MTP off, 4 reps per point, allocation read from the -lv 4 startup log rather than driver counters. Prompt lengths are the server-reported prompt_n, not targets — both models measured at matched occupancy.

Prefill (pp) — no i-quant penalty visible here:

GSQ is ~10% behind on prefill, which is small enough to be the tensor mix rather than an i-quant lookup penalty. Nothing like the slowdown described above.

Decode (tg) — this is where it gets interesting:

UD degrades 66% from shallow to 48k occupancy; GSQ degrades 22%. Both are fully VRAM-resident (GSQ ~12.9 GiB total at 48k, ~3.4 GiB headroom), so this isn't spill. I don't have a mechanism for it — KV layout is identical between the two files, same 16 full-attention layers, same q4_0 KV — but it reproduces at ±1% across reps.

Worth noting the x-axis: most published t/s numbers vary -c with a short prompt, which measures allocated capacity, not how full the KV actually is. Decode speed tracks occupancy, and the two diverge sharply. The table above is matched on server-reported prompt_n so the comparison is apples to apples.

On the K-quant request: I'd hold off on treating the AMD i-quant penalty as universal. It may be real on RDNA2 or on specific backends, but on RDNA3 + ROCm I'm not seeing it, and someone above reports the same on an AMD iGPU with Vulkan. If you do release K-quants, a Q3_K_XL-class build around 11-12 GB would be the useful one for 16GB cards — it leaves room for a large KV plus the MTP head. But if the motivation is purely the prefill concern, it may be worth collecting a couple more measured datapoints first.

Also: the tensor-allocation dumps are genuinely useful. I've been trying to explain a 43% throughput gap between two nominally identical IQ4_XS files from different publishers on this card, and being able to inspect per-tensor assignment without opening the model is exactly what that needs. More publishers should ship those.

@trblnfxn That is an incredible and thorough benchmark breakdown!

However, I should point out a crucial hardware context for my specific case: I am running an AMD RX 6800 (RDNA 2) on Windows using an older/legacy ROCm/HIP build setup.

While RDNA 3 handles newer tensor mixes and I-quants brilliantly (as your data shows), RDNA 2 architectures and legacy Windows ROCm stacks often behave quite differently regarding memory-bound operations and non-uniform quantization lookups. What runs at 100% compute on RDNA 3 / Vulkan iGPUs doesn't always translate identically to RDNA 2 under older Windows toolchains.

This is precisely why a standard Q3_K_XL or Q4_K_M (with MTP) remains heavily requested for the RDNA 2 / 16GB community on Windows, as uniform/K-quant layouts have much more predictable and mature fallback paths on older architectures. Thanks again for sharing those RDNA 3 datapoints—it's super helpful for mapping out the architecture gap!

Fair point — RDNA 2 on a legacy Windows ROCm stack is genuinely a different code path, and I can't speak to it. Worth noting the claim is still unmeasured though: if you have the RX 6800 handy, a pp/tg comparison between one i-quant and one K-quant of the same model at matched prompt length would settle it, and would give ISTA something concrete to allocate against. Right now the K-quant request rests on an architectural expectation rather than a datapoint.

just for reference, on Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf, i can get 1100t/s pp, and gen speed is like between 25-52 t/s decode (when the context usage is above 100k+ it starts to drop around 30t/s decode), on single a RTX3090 with CUDA13.
Here is my llama settings:
-m /home/mert/Models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
--mmproj /home/mert/Models/mmproj-BF16.gguf
-a qwen3.8-27b
--host 0.0.0.0 --port 8080
-ngl 99 -c 200000
--cache-type-k q8_0 --cache-type-v q4_1
--flash-attn on
-b 2048 -ub 1024
-np 1
--cache-ram 4096
--temp 0.6 --min-p 0.05 --presence-penalty 0.0 --top-p 0.95 --top-k 64
--spec-type draft-mtp
--spec-draft-n-max 3
--load-mode mlock
--metrics
--perf
--api-key-file /home/mert/.llama-api-keys
--timeout 600 \

21gb vram usage at these settings.

llama version: 0.4.0-dev (build 302, commit dced507e5)