Qwen 3.8 and Hermes Agent success story
Big up for the capable model.
Unsloth's UD-Q6_K_XL GGUF quant pack on 2x3090 KV@Q8
First impressions we're suspicious due to local benchmarks causing model to overthink for simple tasks.
Currently running model with froggeric chat template, default reasoning on low (I follow noonghunna/club-3090 on github for further evaluation insights for my hardware).
Low reasoning still causes heavy thinking, but the preserve-thinking with adequate prompt cache allows me to break thinking at any time - reacts to in-time steering very well, breaks thinking, accepts updated details and continues higher level thinking. Additionally, hard steering with "STOP" or "Provide output now" help when working in pair-programmer/sysadmin/personal assistance settings, when the thinking is visibly ripe and patience gets challenged.
More reliable than previous models or fine tunes based on qwen35/36 for autonomous agentic work, I used it recently in following tasks: linux server sysadmin (FOSS stack setup, management with ~30 dockerised services, nginx, internal SSO, etc - handles scope very well with additional steering and documentation made available - yet to find a problem it genuinely is incapable of solving in my use case), programming(agentic coding, pair-programming), personal assistance (time management, task aggregation, handling of variety of domains/topics without conflicts/confusion so far). For personal assistance the overthinking on simpler retrieval tasks may turn annoying, hence low reasoning as default.
Also: I enforce my models to follow asd-ste100 standard for simple technical english - cleaner and leaner documentation.
tldr; overthinking model, but controllable with suitable harness and correct workflow/approach - daily driver over alternatives for most tasks.
Thank you for your contribution to the open source community!
Some reference:
Relevant launch params.
MTP: N = 3
CTX: 196k (2^17 + 2^16) [KV: Q8_0]
temp: 0.70
top-p = 0.95
top-k = 20
min-p = 0.02
repeat-penalty = 1.05
presence-penalty = 0.25
reasoning-budget = 20480
Why Q6 instead of INT8? (honest question)
Why Q6 instead of INT8? (honest question)
tldr; Q8 vs Q6 is matter of ctx-size / quality optimisation in given vRAM space for me.
I use Q6 over Q8 (on llama.cpp) to maintain higher context size (190-250k) depending on exact weights and model specifics, additionally it gives me better PP and TPS. With the unsloth quants the perplexity difference between UD-Q6_K_XL vs Q8 variants is neglible.
That being said, from my usage experience, I can tell the difference between Q6_K and Q8_K subtly, there are just some certain silly mistakes / hallucinations the Q6_K occasionally makes over longer reasoning. The Q6_K_XL maintains higher quants in layers where it affects more - Unsloth gives quite detailed overview of their quantisation methods and quite frankly I cannot tell the difference in my use cases between Q8_K and Q6_K_XL (I have not tested Q8_K_XL).
My optimisation is for Hermes agent use case - debugging, minimal scripting, sysadmin on linux, personal assistance, information scraping/aggregation and processing. For complex setup tasks/dev I have been using cloud models in hybrid setting where I deem the local model to be incapable or inefficient. Qwen 3.8 clearly narrowed the gap here between my cloud experts, but is more time consuming.
I find BenchLocal helpful for fast evaluation and confirmation of model benchmarks. To validate benchmarks, as well as quantised variants.