sphaela

/

Qwen3.6-27B-AutoRound-GGUF

like 34

The updated Q2_K_S is impressive in quality with it being able to speak languages other than English without looping or breaking. Even had very good knowledge punching above its model VRAM requirements. Keep it going

I tested the Q2 mixed quant and im really surprised with it how it fits my 3080ti while it stays coherent. Sometimes it gets stuck in a repetitive loop, is there a recommended smapling setting to avoid that for reasoning/general use?

I tested the Q2 mixed quant and im really surprised with it how it fits my 3080ti while it stays coherent. Sometimes it gets stuck in a repetitive loop, is there a recommended smapling setting to avoid that for reasoning/general use?

Hi @BazsiBazsi repetitive loop is a common issue for small models, only settings that might help is the repetitive penalty that you could crank up. I also heard people saying using longer prompt can help but I'm not sure. For your 3080ti if you don't need the vision you could ditch the mmproj and just run the model text only. That way you could fit in more context length. Cheers :3

The new system applies to the Q4KM and Q5KM I noticed they were updated and I just noticed today, should I update as well ?

All the models were updated! Please try them out! Cheers :3

The new system applies to the Q4KM and Q5KM I noticed they were updated and I just noticed today, should I update as well ?

As mentioned above all schemes were re-quantized with the new method, please give them a try!

I wonder, is the Q5KM really near loss-less ?

I wonder, is the Q5KM really near loss-less ?

Honestly there is loss, it's just very small. Q5_K_M uses mixed precision that gives more bits to the sensitive layers (attention) and fewer to the robust ones (FFN), so it punches above its weight for the file size.In practice most people can't tell it apart from full precision in normal use. So I write it that way. It's the recommended scheme if VRAM is not a problem.

I measured the perplexity of your updated Autoround Q4_K_M, Q5_K_M and unsloth's Q5_K_XL(mtp version) using wikitext-2 cut to 1/3 its original size due to my impatience. Here are the results. Autoround Q5_K_M: PPL = 6.2165 +/- 0.06817 Unsloth Q5_K_XL: 6.2297 +/- 0.06882. And yours is even 795MB smaller. This is absolutely crazy.

However, with Q4_K_M I think the autoround bug that caused 35b moe model to output "#" endlessly happened here too. LLamacpp perplexity check failed with "Unexpected negative standard deviation of log(prob)". I tried loading it and it seems to endlessly repeat "#". Can you check the quantization again?

One thing that surprised me, that in my few repeats of svg generation Q5_0 outperformed Q5_K_M, it delivered constantly better images of pelikan riding a bike by a huge degree, tested 3 rounds, not much and its not conclusive by any means, just sharing my experience, it wasnt small difference in quality.

One thing that surprised me, that in my few repeats of svg generation Q5_0 outperformed Q5_K_M, it delivered constantly better images of pelikan riding a bike by a huge degree, tested 3 rounds, not much and its not conclusive by any means, just sharing my experience, it wasnt small difference in quality.

Its PPL is 6.2373 +/- 0.06887, which is noticeably higher than Q5_K_M. I am interested in your observation and will test it right away.

One thing that surprised me, that in my few repeats of svg generation Q5_0 outperformed Q5_K_M, it delivered constantly better images of pelikan riding a bike by a huge degree, tested 3 rounds, not much and its not conclusive by any means, just sharing my experience, it wasnt small difference in quality.

Yeah I can definitely confirm. I wonder why this happens. The prompt was "can you generate an svg of a cyberpunk city"

One thing that surprised me, that in my few repeats of svg generation Q5_0 outperformed Q5_K_M, it delivered constantly better images of pelikan riding a bike by a huge degree, tested 3 rounds, not much and its not conclusive by any means, just sharing my experience, it wasnt small difference in quality.
Yeah I can definitely confirm. I wonder why this happens. The prompt was "can you generate an svg of a cyberpunk city"

This is an interesting find, I wonder, can you try with the Q5KM models before the update sphaela uploaded ?

Q4KM result 10k tokens:

Q5KM result 6k tokens:

Q5_1 result 21k tokens:

Q5KM result (before the quality upgrade):

Well this just shows what iv been observing for some time, svg generation in my opinion shows quite a lot about model capabilities and stability, its not a simple task and the coherence, complexity, details with stable quality can be interpreted as a benchmark on how good model quant is.

I dont care about kld and ppl, thats why i like autoround in the first place, it showed stability and provided correct output more often than other quantization methods. In the end we want correct output in the first place, i dont care if it uses same tokens to produce that or it differs a bit from original if the meaning stays the same.

Well this just shows what iv been observing for some time, svg generation in my opinion shows quite a lot about model capabilities and stability, its not a simple task and the coherence, complexity, details with stable quality can be interpreted as a benchmark on how good model quant is.

I dont care about kld and ppl, thats why i like autoround in the first place, it showed stability and provided correct output more often than other quantization methods. In the end we want correct output in the first place, i dont care if it uses same tokens to produce that or it differs a bit from original if the meaning stays the same.

Unfortunately, I found out q5_0 and q5_1 still performs noticeably behind q5_K_M in coding benchmarks. I asked them "write an html game where you escape a maze with raycasting and wall texture for 3d immersion." and here are the results. They all had the exact same sampling settings(temp=0.1, top k=3 top p=0.5, min p=0.05, repeat penalty=1.1) with no KV cache quantization. Here are zero shot results with thinking enabled with unlimited budget.

Q5_K_M created the entire game in zero shot with no issues.

Q5_0 totally failed with just a laggy static.

Q5_1 was much better but still had weird checkerboard patter on the floor that didn't move how they were supposed to move in 3d.

I ran the test again on Q5_K_M and Q5_1 with 100% greedy sampling to make sure. This time Q5_1 shoved me inside the floor with the game getting like 20 fps while Q5_K_M made mostly working one(however, it placed the escape in unreachable position this time).

Q5_K_M:

Q5_1:

If you want to test the html yourself here are all the files. https://1024terabox.com/s/1aZjFdYA7uhUj1cExNVbL1A

I wonder what would the outcome be in other quantifications of 27b in this coding test, also, did you test with the same seed in each ?

That's interesting, did you set all inference parameters and kv cache same? Also how many times did you repeat the test for each model? I think the result may be highly variable, for example x out of y runs would work, thats why same test needs to be repeated multiple times and aggregate the results.

That’s correct ! I always found better results in coding with autoround models in general, like literally !

I think you should test it with default inference settings --temp 0.6 --min-p 0.0 --top-k 20 --top-p 0.95 because higher temp is needed for creativity and to solve problems that cause model to be stuck. Run at least 3x per each of the tested quants and analyze the results and consistency.

All my tests where done with the coding param suggested by Qwen as you mention it exactly

I think you should test it with default inference settings --temp 0.6 --min-p 0.0 --top-k 20 --top-p 0.95 because higher temp is needed for creativity and to solve problems that cause model to be stuck. Run at least 3x per each of the tested quants and analyze the results and consistency.

I didn't have time for it. I will test it with 3 shot with qwen recommended settings later

I measured the perplexity of your updated Autoround Q4_K_M, Q5_K_M and unsloth's Q5_K_XL(mtp version) using wikitext-2 cut to 1/3 its original size due to my impatience. Here are the results. Autoround Q5_K_M: PPL = 6.2165 +/- 0.06817 Unsloth Q5_K_XL: 6.2297 +/- 0.06882. And yours is even 795MB smaller. This is absolutely crazy.

However, with Q4_K_M I think the autoround bug that caused 35b moe model to output "#" endlessly happened here too. LLamacpp perplexity check failed with "Unexpected negative standard deviation of log(prob)". I tried loading it and it seems to endlessly repeat "#". Can you check the quantization again?

I'm checking this!

I though that 27B was not affected by the upstream underflow bug but Q4_K_M and Q6_K clearly is affected. I'll need to requant these two scheme, sorry for the inconvenience.

I though that 27B was not affected by the upstream underflow bug but Q4_K_M and Q6_K clearly is affected. I'll need to requant these two scheme, sorry for the inconvenience.

Any chances Q5 was also affected by this problem ?

I though that 27B was not affected by the upstream underflow bug but Q4_K_M and Q6_K clearly is affected. I'll need to requant these two scheme, sorry for the inconvenience.

Any chances Q5 was also affected by this problem ?

All Q5 quants passed my test, so I don't think there's any issue.

Thank you sphaela, you're the best! Loving the support you're giving

I'd like to ask you, why is the Q5_1 higher in size than Q5_KM ?

I'd like to ask you, why is the Q5_1 higher in size than Q5_KM ?

The _0/_1 schemes use uniform quantization, every tensor gets the same bit-width (Q5_1 = 5-bit + offset everywhere). The K-quant schemes use mixed precision, they assign higher bits to attention weights (where quality matters more) and lower bits to less sensitive layers.

Q5_K_M gets better perplexity at a smaller size. The _0/_1 variants are legacy formats, K-quants are strictly superior for quality-per-byte. The only reason to use Q5_0/Q5_1 is if your runtime doesn't support K-quants.

This is also why @tigerjjw53 finds the quality difference. For a modern runtime, there's no reason to use uniform quantization models.

Q4_K_M and Q6_K scheme is fixed!

Q4_K_M and Q6_K scheme is fixed!

Thanks for fixing the issue!

Can someone PPL the new Q4KM ? Please 🥹

Can someone PPL the new Q4KM ? Please 🥹

PPL = 6.2542 +/- 0.06934 on the same test set as the q5 variants.

Can someone PPL the new Q4KM ? Please 🥹

PPL = 6.2542 +/- 0.06934 on the same test set as the q5 variants.

wow the difference is 0.6%~ which is pretty much unnoticeable for about 4GB+ of more space

I am experiencing thinking loops with Q4KM @sphaela , first time I encounter this with all your AutoRound models

I am experiencing thinking loops with Q4KM @sphaela , first time I encounter this with all your AutoRound models

sorry to hear about that, but small models like this does prone to thinking loops. I'm cooking something new that might solve this issue :3

As mentioned above all schemes were re-quantized with the new method, please give them a try!

what about Qwen 35B and Gemma 26B?
will you requantize them?

As mentioned above all schemes were re-quantized with the new method, please give them a try!

what about Qwen 35B and Gemma 26B?
will you requantize them?

35B just finished requantize and will be updated today. I'll try to do the Gemma family when I get some compute time!

I had terrible perplexity results, first time trying this, mind sharing your settings?

.\llama-perplexity.exe ^
-m "D:\Local_AI\custom_models\Qwen3.6-27B-MTP-AutoRound-Q3_K_L.gguf" ^
-f "D:\Local_AI\bench_data\wikitext-2-raw\wiki.test.raw"

I also test with more settings, but still terrible result:

.\llama-perplexity.exe ^
-m "D:\Local_AI\custom_models\Qwen3.6-27B-Q2_K_MIXED.gguf" ^
-f "D:\Local_AI\bench_data\wikitext-2-raw\wiki.test.raw" ^
-c 4096 ^
-ngl 99 ^
--split-mode none ^
--main-gpu 1 ^
--fit on ^
--fit-target 0 ^
--flash-attn on ^
--kv-unified ^
--cache-type-k q5_0 ^
--cache-type-v q4_1 ^
--threads 8 ^
--threads-batch 8 ^
--batch-size 2048 ^
--ubatch-size 512