As suggested, I am making a new thread for this issue.
The Qwen3.6-40B-Deck-Opus-NEO-CODE-HERE-2T-OT-HIGH-Q8_0.gguf (which is linked as the q8 version of this model under the model card) seems to loose track and stat outputting nonsense fot me after around 20000 tokens. (it is a version that was dropped 25 days ago).
Anyone know what the problem might be and how to amend it?
The nature of this model is effectively a "brain wipe" then "rehabilitation" to act like Claude. While the model performs better than the base model (Qwen3.6 27B) from my testing, the currently listed gguf files practically require tuning your parameters and system prompt to keep it stable 100% of the time.
Based on what I've seen, the q8_0 variant of this model works well the following settings:
temperature 0.4
top-k 20
top-p 0.95
min-p 0
presence-penalty 0
repeat-penalty 1.1
repeat-last-n 512
cache-type-k f16
cache-type-v f16
flash-attn 'on'
and a strong system prompt.
Notes on fixing gibberish output
My suggestions above prevented all gibberish in my testing, but here are some notes.
The most important thing for this model is a strong system prompt (AGENTS.md). My prompt worked fairly well, which is a derivative of the APEX system prompt.
Use a higher quant (q8_0) when possible. Lower quants are always more likely to generate noise, which will require additional tuning to fix.
cache-type-*: Increase up to f32, if memory allows. The base Qwen models have a known sensitivity to low resolution KV, and that is even more prominent on heavy fine-tunes. While q8_0 is possible, it may require heavier parameter tuning to keep things stable.
top-k: Decrease up to 10. This is the "next character selection pool size". Reducing it limits number of tokens (characters) the model randomly chooses from each time it tries to generate a token. My recommendation is only allow the top 20 "highest probability tokens" to make it into the selection pool, as values below this cutoff have a high chance of inducing gibberish. Reducing it too low can induce thought loops since it would prevent the model from using a lot of it's vocabulary.
Notes on fixing thought loops
repeat-last-n: Increase up to 1024. This applies the repeat-penalty modifier on a phrase repeated within the last repeat-last-n tokens. You should increase this if you see repetitions that are long sequences (e.g. several sentences repeated). If this value is too high (andrepeat-penalty is high as well) then it can induce the model to generate gibberish.
temperature: Increase up to 0.8. Modifies the "randomness" every time the model selects a token from the "next character selection pool". Higher temperatures risk having the model output broken code which it will have to go back to fix. Setting it too low can make the model worse at "creative problem solving". For SW development, this model generates better code at lower temperatures.
repeat-penalty: Increase up to 1.15. To give an example, a value of 1.1 puts a small (10%) inclination for the model to not repeat phrases (within the last repeat-last-n number of tokens). If you set it too high then it can induce the model to generate gibberish or terminate responses early.
Note to maintainer (DavidAU)
I assume adding an extra training step specifically for long-context (200k to 260k) reasoning could alleviate the hard requirement to tune these settings? While the base model (Qwen3.6 27B) does have a known tendency to form thought loops (with default llama.cpp config), the slight chance of gibberish as the context grows seems to be unique to this fine-tune.
