18GB varaint?
Great work! I'm just trying out this!
A varaint with around 18 GB file size maybe could be intersting, too. Maybe a bigger Q4 varaint or a smaller Q5.
Can you do this?
so something like unsloth does with the Q4_XL variants?
exactly!
What are your memory requirements? Because the IQ4_XS I produced is of a better quality than the normal IQ4_XS. I don't know if there would be any need to just have another model file that is half or one gigabyte larger and there will be no real quality difference.
I think it would be good if there were a Q4_K_M version.
What are your memory requirements? Because the IQ4_XS I produced is of a better quality than the normal IQ4_XS. I don't know if there would be any need to just have another model file that is half or one gigabyte larger and there will be no real quality difference.
OK so I did a Q4_K_M version of it, but I don't know why it got flagged unsafe by huggingface, its a normal Q4 quant. I'll send them an email.
Thank you!
I'll try it out!
I'll run it on 28GB VRAM (16+12).
Yesterday I figured out my llama.cpp build with TQ and MTP doesn't perform well. I'm using the atomic-llama-cpp-turboquant fork. Does somebody have a better alternative?
Try to use the mainline one. they have also some new fixes for MTP. turboquant is overrated, and you can use Q4_0 with great results. I also use it with Q4 KV cache and it goes very well.
Currently in fact I use mainline with a build from yesterday (with Q8 KV on the 27B model, qwen3.6 35B-A3B fits easily on native KV).
I experience a big degration in accuracy (not measured, only my feeling/experience) as soon as my context fills up to ~50k. For agentic coding I need at least 100k useable "good" context. That's why I want best possible model and KV accuracy I can get.