This model keeps quitting mid-way and/or doesn't answer
I'm going back to unsloth I guess.
Here is reasoning/thinking:
What are the prompts or queries to reproduce this behavior?
Many different types, but the screenshots are part of my internal benchmarks. The most I've encountered this is over long horizon coding tasks on Pi, though I did spend more time on long horizon coding tasks. Coding can be for Python, frontend, etc.
I am also experiencing this issue. The quantization delivers excellent quality when functioning normally, but it occasionally halts unexpectedly mid-generation.
In two recent instances, it exited the reasoning phase on the word have, generated a few words of content output, and then stopped entirely right after the word not.
My setup uses Java with direct OpenAI API calls (via a custom agent framework).
I usually find when a model is fully coherent and cuts off half way through a sentence, a max token limit is to blame. At least in the first screenshot, hover over the i in the circle. Should tell you details about how many tokens are being used. See if it always fails around the same number of tokens.
I usually find when a model is fully coherent and cuts off half way through a sentence, a max token limit is to blame. At least in the first screenshot, hover over the i in the circle. Should tell you details about how many tokens are being used. See if it always fails around the same number of tokens.
That's a really good point and a likely cause.
If anyone is running into that issue, could you check whether it is terminated by hitting max length?
In my case, it wasn't the max token limit or a full context window (e.g., context was ~90k out of 262k, with a relatively short response). The model actually recovered easily and proceeded to a full answer once I continued, keeping all previous messages (including reasoning) and adding a new message noting that the response was cut off.
Stopping prematurely is a known LLM behavior, similar to hallucinations or repetition loops. It's too early for me to say for sure whether this quantization suffers more from it, though I suspect it might. Still, most of the time it performs very well. If it does happen more often here, perhaps post-tuning or quantization alignment somehow encouraged premature <eos> (End-of-Sequence) generation?
I usually find when a model is fully coherent and cuts off half way through a sentence, a max token limit is to blame. At least in the first screenshot, hover over the i in the circle. Should tell you details about how many tokens are being used. See if it always fails around the same number of tokens.
That's a really good point and a likely cause.
If anyone is running into that issue, could you check whether it is terminated by hitting max length?
Definitely not because of hitting max length. The screenshot in particular was at 100s of tokens. I set the max to 262k.