unsloth

/

Qwen3.8-27B-GGUF

like 1.4k

Follow

Unsloth AI 31.3k

I'm also getting about 7 t/s
I get about 20 t/s for Qwen 3.6

I assume it's because 3.6 used a MoE (mixture of experts) and 3.8 is a dense model.

I'm also getting about 7 t/s
I get about 20 t/s for Qwen 3.6

I assume it's because 3.6 used a MoE (mixture of experts) and 3.8 is a dense model.

Qwen 3.6 27B is a dense model, just like Qwen 3.8 27B. Moreover, they share absolutely similar architecture. Only Qwen 3.6 35B A3B based on MoE architecture, unlike these two.

I tested quantizations UD-Q4_K_XL and Q4_K_M on CMP 50HX (upgraded to 20GB VRAM) with llama.cpp b10434. No matter of context size, KV-cache quantization, enabled/disabled MTP: performance always is around 7-10 t/s. Qwen 3.6 27B Q4_K_M with enabled MTP gives around 25 t/s on same hardware.
How did you mod to 20GB for 50Hx, people telling me it's very difficult because they need a new PCB for it?

@rosspanda0 you don't need for new PCB. There is some videos on Youtube: https://www.youtube.com/watch?v=DhHYhkx8RiQ (english) https://www.youtube.com/watch?v=MPJkxq5kNDM (russian).
And patch for llama.cpp to increase speed of generation - https://github.com/ggml-org/llama.cpp/issues/24616#issuecomment-5246453748
Also, you can solder 24 smd and get PCI 1@x16

For electricity saving on CMP50-90 everyone need to install https://github.com/sasha0552/nvidia-pstated

My numbers on 3xCMP50HX (20gb,10gb,10gb)
prompt processing, n_tokens = 4665, progress = 1.00, t = 50.01 s / 93.28 tokens per second
n_decoded = 5078, tg = 33.02 t/s, tg_3s = 32.90 t/s

GGML_CUDA_GRAPH_OPT=1 CUDA_VISIBLE_DEVICES=2,1,3 ./llama-server -m /home/llama/models/Qwen3.8-27B-Q6_K.gguf -c 72000 -fa on --host 0.0.0.0 --port 8080 --jinja --threads 12 --temp 0.75 --top-p 0.95 --top-k 40 --min-p 0.05 --presence-penalty 0.25 --repeat-penalty 1.05 --no-context-shift -b 6000 -ub 6000 --no-mmap --threads-batch 12 --metrics -fit on --reasoning off -np 1 --cache-ram 24000 --parallel 1 --swa-checkpoints 64 --ctx-checkpoints 64 --swa-full -fitt 222 --kv-unified -cb -fa 1 --keep -1 -mg 0 -ts 20,10,6 --spec-type draft-mtp --spec-draft-n-max 3

@rosspanda0 you don't need for new PCB. There is some videos on Youtube: https://www.youtube.com/watch?v=DhHYhkx8RiQ (english) https://www.youtube.com/watch?v=MPJkxq5kNDM (russian).
And patch for llama.cpp to increase speed of generation - https://github.com/ggml-org/llama.cpp/issues/24616#issuecomment-5246453748
Also, you can solder 24 smd and get PCI 1@x16

For electricity saving on CMP50-90 everyone need to install https://github.com/sasha0552/nvidia-pstated

My numbers on 3xCMP50HX (20gb,10gb,10gb)
prompt processing, n_tokens = 4665, progress = 1.00, t = 50.01 s / 93.28 tokens per second
n_decoded = 5078, tg = 33.02 t/s, tg_3s = 32.90 t/s

GGML_CUDA_GRAPH_OPT=1 CUDA_VISIBLE_DEVICES=2,1,3 ./llama-server -m /home/llama/models/Qwen3.8-27B-Q6_K.gguf -c 72000 -fa on --host 0.0.0.0 --port 8080 --jinja --threads 12 --temp 0.75 --top-p 0.95 --top-k 40 --min-p 0.05 --presence-penalty 0.25 --repeat-penalty 1.05 --no-context-shift -b 6000 -ub 6000 --no-mmap --threads-batch 12 --metrics -fit on --reasoning off -np 1 --cache-ram 24000 --parallel 1 --swa-checkpoints 64 --ctx-checkpoints 64 --swa-full -fitt 222 --kv-unified -cb -fa 1 --keep -1 -mg 0 -ts 20,10,6 --spec-type draft-mtp --spec-draft-n-max 3

awesome! I'm watching the russian video

Same experience here - I have dual r9700's and I get ~52 TPS with Qwen 3.6 27B, but I'm only getting ~6 TPS in Qwen 3.8 27B with practically identical LLama.cpp settings

I had this same experience but realized I had also downloaded the mmproj file, after deleting it, the model stopped offloading to the CPU
I'm getting ~57 tokens/sec using an RTX Pro 4000 Blackwell.
My settings are
~/llama.cpp/build/bin/llama-server
--host 127.0.0.1
--port 8080
--alias Qwen3.8-27B
--model /mnt/ai/models/Qwen3.8-27B/Qwen3.8-27B-Q4_K_M.gguf
--jinja
--metrics
--spec-type draft-mtp
--spec-draft-n-max 3
--n-gpu-layers auto
--n-predict 32768
--ctx-size 131584
--batch-size 2048
--ubatch-size 512
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn auto
--kv-unified
--fit on
--load-mode mlock
--parallel 1
--threads 12
--threads-batch 12
--min-p 0.0
--temperature 0.7
--top-k 20
--top-p 0.95
--repeat-penalty 1.0
--presence-penalty 1.5

Metric | Value | Rate
Prompt eval | 2625 tokens / 3068.30 ms | 855.52 tok/s (1.17 ms/tok)
Generation | 1264 tokens / 22191.39 ms | 56.91 tok/s (17.57 ms/tok)
Total | 3889 tokens / 25259.69 ms | —
CUDA graphs reused | 709 | —
Spec decoding acceptance | 0.8274 (901/1089), mean len 3.48 | —

2x 5060 ti 16gb's getting over 60 tps on q5km with mtp enabled. I am using some ancient ollama from 2-3 months ago.