loopy, hallucinates and tool calling problems
trying that model via llama.cpp, download fork llama.cpp and build it.
i got 4070 ti 12gb, using in vscode via copilot chat, continue, cline, opencode, roo code, zoo code, but its not working properly.
I've set recommended params and other params, but, its loopy (on reasoning, tool calling), hallucinates and got tool calling problems (in response i got and its not works).
changed temperature, presence penalty, repeat penalty, top-k, kv cache types. Unfortunately it doesn't work and it's very disappointing that it doesn't work as it should. I've hoped that i can run 27b dense model in my 4070ti
Try Gemma 4 26B A4B in Q2_K_XL - running perfect on my 4070 with 110k context (in Q4) - these Bonsai Models are nice, but they are not trained in 1bit but compressed to hell. Check Microsofts BitNet and pray someone will train a Model with lets say 50B parameter with that framework. That would be revolutionary, not this compressed Qwen toy. But the direction is correct, 1 bit is the future.
Try Gemma 4 26B A4B in Q2_K_XL - running perfect on my 4070 with 110k context (in Q4) - these Bonsai Models are nice, but they are not trained in 1bit but compressed to hell. Check Microsofts BitNet and pray someone will train a Model with lets say 50B parameter with that framework. That would be revolutionary, not this compressed Qwen toy. But the direction is correct, 1 bit is the future.
tried, its loopy too. You running on llama.cpp? can you share your start params
on my pc works perfect qwen3.6-35b-a3b moe.
current q2 bonsai not great for agentic, may be you can try with tool calling model like needle 26m as extension. but to be honest i prefer ornith 1.0 9b compared to this heavily quantized 27b as i have no issues in opencode when i ran on my 3060 12gb.
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
Same for me, never works, also stuck in loop. however original qwen3.6 s7b Q6 never have this problem, I am using LM studio on Mac with MLX version
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
jesus. why did you feel you need to quant kv cache to q4 when it's already in q2? by doing that you were downgrading q2 to q0.5?! and it still worked good enough for you to feel that's it?!
I feel way more confident in this model now.
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
jesus. why did you feel you need to quant kv cache to q4 when it's already in q2? by doing that you were downgrading q2 to q0.5?! and it still worked good enough for you to feel that's it?!
I feel way more confident in this model now.
bcs i before dont have enough vram...
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
jesus. why did you feel you need to quant kv cache to q4 when it's already in q2? by doing that you were downgrading q2 to q0.5?! and it still worked good enough for you to feel that's it?!
I feel way more confident in this model now.bcs i before dont have enough vram...
offload it to cpu
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
jesus. why did you feel you need to quant kv cache to q4 when it's already in q2? by doing that you were downgrading q2 to q0.5?! and it still worked good enough for you to feel that's it?!
I feel way more confident in this model now.bcs i before dont have enough vram...
I understand your pain brother, chunk your tasks as best as you can if you're trying to keep things fully local, offload to CPU as user Imboard mentioned and keep your head held high.
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
jesus. why did you feel you need to quant kv cache to q4 when it's already in q2? by doing that you were downgrading q2 to q0.5?! and it still worked good enough for you to feel that's it?!
I feel way more confident in this model now.
Model weight quant is different from kv cache quant, you can't just multiply them like that
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
jesus. why did you feel you need to quant kv cache to q4 when it's already in q2? by doing that you were downgrading q2 to q0.5?! and it still worked good enough for you to feel that's it?!
I feel way more confident in this model now.Model weight quant is different from kv cache quant, you can't just multiply them like that
while that is true mathematically, I'm simplifying the math to make it more understandable, you are however feeding lower tail kv cache to a model with extremely low tail (practically no tail), which is in effect around ~0.6q.
the model runs better even with q4 kv cache, when i used this template and dry sampling.
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
--chat-template-file chat_template.jinja ^
--dry-multiplier 0.8 ^
--dry-base 1.75 ^
--dry-allowed-length 2 ^
--dry-penalty-last-n 256 ^
about looping and tool call problems, i startup ternary bonsai Q2 on gpu with 20gb vram and dont select kv cache q4 or q8, set it as default f16, looping is gone and tool calling works perfect. Continue testing.
jesus. why did you feel you need to quant kv cache to q4 when it's already in q2? by doing that you were downgrading q2 to q0.5?! and it still worked good enough for you to feel that's it?!
I feel way more confident in this model now.
Bro, that was - simply wrong. But you already got this corrected :)