Tool use failure [Fix Found]
The model performance and reasoning seems pretty fantastic, but the tool use has been absolutely impossible to use in an agentic mode with LM Studio and OpenCode. It loops non-stop sending the wrong arguments for file writes, yet if I make a quick custom MCP server it successfully uses those tools. I've not seen any model fail to work with the tools in OpenCode yet. Is there something I'm missing? (I don't pretend to be an expert)
Actually, enabling preserve_thinking fixed the tool issues, at least for me... There was also an issue with OpenCode's max token output which was causing it to cut the stream off mid-way resulting in arguments for the tool calls being chopped. After enabling preserve_thinking and raising the OpenCode output token limit, it's working perfectly.
Really amazing model - and thank you so much for it.
Really thank for you sharing, you totally saved my day!
Preserve thinking makes a difference for sure. I'm also finding that running the 6_K or 8_0 quantizations improve the stability of tool use for this model compared to 4_K_M. However using the more granular quantizations may not be realistic for most consumer hardware especially at longer contexts
Is preserve thinking something you set in opencode, in llama.cpp or both?
It’s set in the chat template. Some runners like LM Studio have a checkbox for it under the inference section. Otherwise just set preserve_thinking = true in the jinja template.
For the discussion on quants: without preserve thinking I’ve seen the same exact behavior on all quants. I’m running both q4 and q6 because I’m testing whether or not there really is a big loss between them and so far I’ve not seen one. I’ve seen q6 perform a bit worse at higher contexts than q4 oddly, with q4 producing absolutely zero tool call failures and with q6 producing a few here and there that it self-corrects and then calls with the right arguments. For me, so far, q4 has been the winner between q4 and q6… q8 has been decent but I’ve not seen enough of any difference between 6 and 8 to justify using it.
For dense 27b I’ve seen a massive difference between the quants in terms of both efficacy of nailing something on first pass versus having to nudge it. Of course, that comes with the performance penalties of running a dense model. I’ve not seen good enough results to justify it. The models raw knowledge versus its CoT and reasoning are two different things.
Hmm.... That is weird, bacause I'm getting slowly insane by the fact that like 1/3 of the time the tool calling on this thing does not work. And when used in longer agentic work a wrong tool call right away crashes the session quite often 😡
One thing I notice is these models really struggle with moderately long path names, and they really, really want to use the full path name for a tool call even when they are sitting in the directory with the file they need to work on. The most frustrating part is seeing it perfectly name the path in its thinking, then when it makes the tool call there's an extra space or missing letter, causing it to fail. It's also entirely possible a slight tweak to my generation parameters could fix it, but I haven't found the perfect settings so far.
This has led me to building a Pi extension that can manage the tool calls and minimize context pollution from bad outputs and failed tool calls. The whole thing is pretty ridiculous and probably overly complicated, but it's pretty fun watching it (mostly) work.
