Share your model speed here
Hey everyone!
I think this topic could be really useful for users who have plenty of system RAM to run larger models, but are limited by GPU VRAM
To make comparisons easier and help others understand what models/settings might work for their hardware, please share your setup and performance using the format below:
- 1. Model file / model name
- 2. GPU: model + VRAM
- 3. CPU: model
- 4. RAM: size + type
- 5. Generation speed: tokens/sec
- 6. App / frontend: e.g. FreeToken, llama.cpp, LM Studio, etc.
- 7. Launch parameters / settings:
- 8. Extra notes: quantization, context size, offloading, optimizations, or anything else worth mentioning
This should make it much easier for people to compare setups and decide which quantization to download
Curious to see if my old M1 ultra will have enough juice to run GGUF.
Also surprised that GGUF are already being uploaded, I thought this was a new architecture. Does unsloth have a llama.cpp fork that can already run this?
Im hoping I can fit this in my AMD Ryzen Ai 395+ 128Gb sadly my 3 x 4090 will be useless for this LLM
It will fit on a 128GB UMA, just a matter quantization. Unsloth writes the 4-bit will be 110GB, so that might be a bit tight on 128GB, hopefully we can offload ngram to SSD (or stream them or whatever its called).
AI MAX+ 395 gfx1151 rocm7.14
Qwen3.8-Flash-Next-UD-IQ1_S
PP ~200-300t/s
TG ~20t/s
Edited: Q4_K_XL
Not scientific benchmarks yet (just seeing the llama-server numbers as I test it)
M1 Ultra 128G
Qwen3.8-Flash-Next-UD-IQ1_S
PP ~ 400 tps
TG ~ 20 tps
Just a quick one before hitting bed:
AI MAX+ 395 gfx1151 + R9700 gfx1201
Qwen3.8-Flash-Next-UD-IQ4_XS
llama-bench --model /ai/models/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf -p 512,4096 -n 128,1024 -d 0,512 --device Vulkan0/Vulkan1 -ngl 99 -fa 1 -ts 40/50 --split-mode layer
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model | size | params | backend | ngl | fa | dev | ts | test | t/s |
|---|---|---|---|---|---|---|---|---|---|
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | pp512 | 532.87 ± 11.64 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | pp4096 | 464.10 ± 4.44 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | tg128 | 23.25 ± 0.10 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | tg1024 | 23.03 ± 0.47 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | pp512 @ d512 | 497.23 ± 14.41 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | pp4096 @ d512 | 462.56 ± 7.05 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | tg128 @ d512 | 23.16 ± 0.46 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | Vulkan | 99 | 1 | Vulkan0/Vulkan1 | 40.00/50.00 | tg1024 @ d512 | 22.15 ± 0.22 |
build: 035e22731 (10656)
Qwen3.8-Flash-Next-UD-Q4_K_XL
CPU: 2x Xeon 2697v4
GPU: 2x 7800XT + Mi50 32Gb + 4Mi50x16Gb (128GB) + 2x Nvidia P100 (unused)
Backend: llama.cpp
Split mode: layer
PP 288 / TG 16 @CTX4K
Just a fast test, waiting for further optimizations, MiniMax M2.7 Q3_K 110Gb runs at PP 400 / TG 20
CUDA_VISIBLE_DEVICES=999 HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 GGML_CUDA_DISABLE_GRAPHS=1 GGML_CUDA_GRAPH_OPT=0 ./llama-server-qw4 -m /mnt/md4t/llms/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 60000 --host 0.0.0.0 --port 8081 --threads 12 --temp 0.65 --top-p 0.95 --top-k 40 --min-p 0.05 --presence-penalty 0.25 --repeat-penalty 1.05 --no-context-shift -b 2200 -ub 2200 --load-mode none --threads-batch 24 -fa on -fit on --reasoning off -np 1 --cache-ram 32000 --parallel 1 --swa-checkpoints 64 --ctx-checkpoints 64 --swa-full --kv-unified -cb --keep -1 -mg 0 --jinja --chat-template-file /home/llama/models/Qwen3.8-27B-UD-Q6_K.chat_template.jinja -fitt 128
///////
UPDATE AFTER OFFICIAL MERGE
//////
prompt eval time = 17410.47 ms / 4685 tokens ( 3.72 ms per token, 269.09 tokens per second)
eval time = 245548.29 ms / 4763 tokens ( 51.56 ms per token, 19.39 tokens per second)
Thank you Unsloth for zero day access and your work!
edit: After some tinkering around i could squeeze a few more tokens out of it, looking forward to more optimizations and projects like: https://github.com/syv-ai/qwen38-27b-rtx3090 for this model.
- Model: Qwen3.8-Flash-Next-UD-Q4_K_XL
- GPU: RTX 3090TI 24GB
- CPU: Ryzen 9 5950X C16/T32
- RAM: 128GB DDR4 3200mhz
- Generation speed with 30k prompt: 15t/s | Processing speed: ~340t/s | Eval Speed: ~310t/s
- Engine & Frontend: llama.cpp with this fix https://github.com/ggml-org/llama.cpp/pull/27742
- Launch Parameter: llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL.gguf --host 127.0.0.1 --port 8282 --device CUDA0 --ctx-size 130000 --fit on --fit-target 1024 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --spec-type ngram-mod --batch-size 2048 --ubatch-size 2048 --threads 12 --load-mode none --metrics --alias Qwen3.8-Flash-Next-UD-Q4_K_XL
- Context Size: 130k
I have a AI MAX+ 395 gfx1151 running in fedora with rocm 7.1
I've put Qwen3.8-Flash-Next-UD-IQ4_XS to do some real agentic work and it's decent (very similar performance to qwen3.8 27b)
8.10.118.425 I slot print_timing: id 2 | task 2248 | prompt eval time = 3548.50 ms / 517 tokens ( 6.86 ms per token, 145.70 tokens per second)
8.10.118.428 I slot print_timing: id 2 | task 2248 | eval time = 95902.06 ms / 1070 tokens ( 89.71 ms per token, 11.15 tokens per second)
8.10.118.429 I slot print_timing: id 2 | task 2248 | total time = 99450.56 ms / 1587 tokens
on my side I smell something's fishy with the current llama.cpp PR. cf. https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/discussions/22#6a900962d7cc2416049c53c7
Qwen3.8-Flash-Next UD-IQ3_XXS — RTX 5070 Ti 16GB + RTX 5060 Ti 16GB / 64GB RAM
Got Qwen3.8-Flash-Next UD-IQ3_XXS running on a dual 16GB consumer GPU setup with only 64GB system RAM.
System
- GPU: RTX 5070 Ti 16GB + RTX 5060 Ti 16GB
- CPU: Ryzen 7 9700X
- RAM: 64GB
- OS: Windows
- Backend: llama.cpp
- llama.cpp: PR #27742
- Quant: UD-IQ3_XXS
- Context: 32K
- KV cache: F16
- MTP: Disabled
- Speculative decoding: Disabled for this benchmark
Launch parameters
llama-server.exe ^
-m "Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf" ^
--host 127.0.0.1 ^
--port 18010 ^
-c 32768 ^
-fa on ^
-ctk f16 ^
-ctv f16 ^
-b 1024 ^
-ub 256 ^
-t 8 ^
-tb 12 ^
-np 1 ^
--jinja ^
--fit on ^
--fit-target 1536 ^
-sm layer
Benchmark
Benchmark methodology:
repeat = 3
discard-first = 2
warmup enabled
Average decode: 23.32 tok/s
| Workload | Decode |
|---|---|
| Code | 23.55 tok/s |
| Structured | 23.46 tok/s |
| Prose | 23.18 tok/s |
Long-context needle test
- Prompt: 5,494 tokens
- Prefill: 216.48 tok/s
- Decode: 22.22 tok/s
Overall benchmark
- Average prompt processing: 86.87 tok/s
- Average decode: 23.32 tok/s
- End-to-end aggregate: 18.02 tok/s
- Generated: 3,803 tokens
- Wall time: 211.07s
Full benchmark results
prompt grp p_tok prompt t/s pred t/s
--------------------- ------ ----- ---------- --------
code_python code 72 62.84 23.19
code_cpp code 82 67.67 23.49
explain_concept prose 69 62.31 23.52
summarize prose 104 78.51 23.54
qa_factual prose 66 51.92 21.57
translation prose 67 65.32 23.59
creative_short prose 63 64.45 24.10
stepwise_math code 102 80.73 23.67
json_output struct 122 80.04 23.42
long_reasoning prose 111 80.55 23.60
repeat_pattern struct 103 77.76 23.45
code_completion code 168 97.99 23.84
long_code_review prose 773 179.45 23.18
structured_sql_schema struct 85 68.34 23.09
high_entropy_creative prose 76 62.22 23.02
long_context_needle prose 5494 216.48 22.22
tool_calling_format struct 111 80.12 23.89
Notes
--fit on works much better on this setup than manually forcing CPU MoE offload.
I tested manual placement with:
--override-tensor "per_layer_token_embd=CPU"
-ngl 28
-ncmoe 32
It fit in memory, but decode performance dropped to roughly 8–11 tok/s, versus ~23 tok/s with automatic fit.
I also tested different --fit-target values:
- 3072 MiB: ~19–20 tok/s steady-state
- 2048 MiB: ~19–21 tok/s
- 1536 MiB: ~20–23 tok/s
So far 1536 MiB with automatic fit works best for me.
F16 KV is used because the current quantized KV path is still experimental. No MTP yet.
Pretty impressed that an ~82GB GGUF can run at around 23 tok/s on 64GB system RAM + 2×16GB consumer GPUs with heavy offloading.
Spec- ryzen 9 5950x, 64GB ddr4, 2x RTX 3080 10GB (total 20GB vram)
offloaded n-gram to nvme ssd , and getting 16-17t/s on Q4_k_XL
Qwen3.8-Flash-Next-UD-Q4_K_XL
RTX 6000 PRO 96GB + RTX 5090
I9 14900KF
48gb DDR5 8000MHZ
60 T/S at first going down to 30 T/S after 128k context.
llama.cpp
--host 0.0.0.0 ^
--port 8001 ^
--ctx-size 262144 ^
--n-gpu-layers 99 ^
--parallel 1 ^
--flash-attn on ^
--reasoning on ^
--chat-template-kwargs "{"reasoning_effort":"xhigh"}" ^
-t 24 ^
-lv 4 ^
--fit off ^
--alias Qwen3.8-Flash-Next ^
--split-mode layer ^
--tensor-split 3,1 ^
--override-tensor "per_layer_token_embed=CUDA0,ple_ngram_embed=CUDA0" ^
--load-mode none ^
--cache-prompt ^
--metricsthe vram is not utilized, leaving 32 VRAM free for some reason.
Qwen3.8-Flash-Next-UD-Q4_K_XL
RTX 6000 PRO 96GB + RTX 5090
I9 14900KF
48gb DDR5 8000MHZ
60 T/S at first going down to 30 T/S after 128k context.
llama.cpp
--host 0.0.0.0 ^
--port 8001 ^
--ctx-size 262144 ^
--n-gpu-layers 99 ^
--parallel 1 ^
--flash-attn on ^
--reasoning on ^
--chat-template-kwargs "{"reasoning_effort":"xhigh"}" ^
-t 24 ^
-lv 4 ^
--fit off ^
--alias Qwen3.8-Flash-Next ^
--split-mode layer ^
--tensor-split 3,1 ^
--override-tensor "per_layer_token_embed=CUDA0,ple_ngram_embed=CUDA0" ^
--load-mode none ^
--cache-prompt ^
--metricsthe vram is not utilized, leaving 32 VRAM free for some reason.
Nice setup, not completely sure but maybe --fit on will fix that, also try lowering your threads to 16 or 12 could improve performance as well.
Qwen3.8-Flash-Next-UD-Q4_K_XL
RTX 6000 PRO 96GB + RTX 5090
I9 14900KF
48gb DDR5 8000MHZ
60 T/S at first going down to 30 T/S after 128k context.
llama.cpp
--host 0.0.0.0 ^
--port 8001 ^
--ctx-size 262144 ^
--n-gpu-layers 99 ^
--parallel 1 ^
--flash-attn on ^
--reasoning on ^
--chat-template-kwargs "{"reasoning_effort":"xhigh"}" ^
-t 24 ^
-lv 4 ^
--fit off ^
--alias Qwen3.8-Flash-Next ^
--split-mode layer ^
--tensor-split 3,1 ^
--override-tensor "per_layer_token_embed=CUDA0,ple_ngram_embed=CUDA0" ^
--load-mode none ^
--cache-prompt ^
--metricsthe vram is not utilized, leaving 32 VRAM free for some reason.
Nice setup, not completely sure but maybe --fit on will fix that, also try lowering your threads to 16 or 12 could improve performance as well.
thanks, i tried that but no change, actually loading only on the 6000 pro was a bit better for some reason.
Qwen3.8-Flash-Next-UD-Q4_K_XL
RTX 6000 PRO 96GB + RTX 5090
I9 14900KF
48gb DDR5 8000MHZ
60 T/S at first going down to 30 T/S after 128k context.
llama.cpp
--host 0.0.0.0 ^
--port 8001 ^
--ctx-size 262144 ^
--n-gpu-layers 99 ^
--parallel 1 ^
--flash-attn on ^
--reasoning on ^
--chat-template-kwargs "{"reasoning_effort":"xhigh"}" ^
-t 24 ^
-lv 4 ^
--fit off ^
--alias Qwen3.8-Flash-Next ^
--split-mode layer ^
--tensor-split 3,1 ^
--override-tensor "per_layer_token_embed=CUDA0,ple_ngram_embed=CUDA0" ^
--load-mode none ^
--cache-prompt ^
--metricsthe vram is not utilized, leaving 32 VRAM free for some reason.
Nice setup, not completely sure but maybe --fit on will fix that, also try lowering your threads to 16 or 12 could improve performance as well.
thanks, i tried that but no change, actually loading only on the 6000 pro was a bit better for some reason.
Try removing the override-tensor flag and use --device CUDA0,CUDA1 maybe that will help
Qwen3.8-Flash-Next-UD-Q4_K_XL
AMD EPYC 7763 + DDR4 RAM
(some cmd line params are doubled because of how I generate them. just keep the latest one)
1st config : RTX 6000 PRO 96GB WS @450W (during TG the card stays at 70% usage and 300W)
llama-server -lv 4 --model /storage/llms/models/GGUF-Qwen3.8-Flash-Next-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --port 5802 -b 8192 -ub 1024 --cont-batching --jinja -ngl 99 -c 262144 --load-mode none --flash-attn on --parallel 1 --metrics -ctk q8_0 -ctv q8_0 --mmproj /storage/llms/models/GGUF-Qwen3.8-Flash-Next-UD-Q4_K_XL/mmproj.gguf --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --no-host --override-tensor 'per_layer_token_embd.weight=CPU' --flash-attn auto --reasoning-preserve --no-mmproj-offload
0.00.695.945 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
0.00.695.950 I common_memory_breakdown_print: | - CUDA0 (RTX PRO 6000 Blackwell Workstation Edition) | 97249 = 95850 + (87491 = 78056 + 4600 + 4834) + -86092 |
0.00.695.951 I common_memory_breakdown_print: | - Host | 31750 = 28110 + 0 + 3640 |
2.20.615.511 I slot print_timing: id 0 | task 0 | prompt eval time = 8868.66 ms / 22695 tokens ( 0.39 ms per token, 2559.01 tokens per second)
2.20.615.515 I slot print_timing: id 0 | task 0 | eval time = 66848.76 ms / 3992 tokens ( 16.75 ms per token, 59.70 tokens per second)
2.20.615.516 I slot print_timing: id 0 | task 0 | total time = 75717.42 ms / 26687 tokens
2nd config : 2x RTX 4090D modded 48GB each @300W (boths cards stays at 35% / 130W)
llama-server -lv 4 --model /storage/llms/models/GGUF-Qwen3.8-Flash-Next-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --port 5803 -b 8192 -ub 1024 --cont-batching --jinja -ngl 99 -c 262144 --load-mode none --flash-attn on --parallel 1 --metrics -ctk q8_0 -ctv q8_0 --split-mode layer -ts 1,1 -fit off --mmproj /storage/llms/models/GGUF-Qwen3.8-Flash-Next-UD-Q4_K_XL/mmproj.gguf --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --no-host --override-tensor 'per_layer_token_embd.weight=CPU' --flash-attn auto --reasoning-preserve --no-mmproj-offload
5.27.591.167 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
5.27.591.171 I common_memory_breakdown_print: | - CUDA0 (RTX 4090 D) | 48510 = 1034 + (46711 = 40277 + 2303 + 4130) + 763 |
5.27.591.171 I common_memory_breakdown_print: | - CUDA1 (RTX 4090 D) | 48510 = 3576 + (44168 = 37778 + 2296 + 4092) + 765 |
5.27.591.171 I common_memory_breakdown_print: | - Host | 31750 = 28110 + 0 + 3640 |
4.17.359.940 I slot print_timing: id 0 | task 0 | prompt eval time = 12630.69 ms / 22695 tokens ( 0.56 ms per token, 1796.81 tokens per second)
4.17.359.945 I slot print_timing: id 0 | task 0 | eval time = 178790.66 ms / 7986 tokens ( 22.39 ms per token, 44.66 tokens per second)
4.17.359.945 I slot print_timing: id 0 | task 0 | total time = 191421.36 ms / 30681 tokens
19 t/s decoding
65 t/s prompt processing
Intel i9 12900k
128 GB DDR4 3600 MT/s
RTX 3090
Manjaro Linux
llama-server
--model Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--alias qwen3.8-flash-next-local
--ctx-size 262144
--parallel 1
--n-gpu-layers all
--n-cpu-moe 41
--override-tensor per_layer_token_embd=CPU
--fit off
--no-op-offload
--no-repack
--no-host
--load-mode mmap
--flash-attn on
--cache-type-k q8_0
--cache-type-v q8_0
--batch-size 2048
--ubatch-size 128
--threads 8
--cpu-mask 0x5555
--cpu-strict 1
--threads-batch 24
--cpu-range-batch 0-23
--cpu-strict-batch 1
--cache-ram 0
--no-cache-idle-slots
--jinja
--reasoning on
--reasoning-effort xhigh
--reasoning-preserve
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0
--presence-penalty 0
--repeat-penalty 1.0
--host 127.0.0.1
--port 8080
--cors-origins localhost
Full benchmark results
prompt grp p_tok prompt t/s pred t/s --------------------- ------ ----- ---------- -------- code_python code 72 62.84 23.19 code_cpp code 82 67.67 23.49 explain_concept prose 69 62.31 23.52 summarize prose 104 78.51 23.54 qa_factual prose 66 51.92 21.57 translation prose 67 65.32 23.59 creative_short prose 63 64.45 24.10 stepwise_math code 102 80.73 23.67 json_output struct 122 80.04 23.42 long_reasoning prose 111 80.55 23.60 repeat_pattern struct 103 77.76 23.45 code_completion code 168 97.99 23.84 long_code_review prose 773 179.45 23.18 structured_sql_schema struct 85 68.34 23.09 high_entropy_creative prose 76 62.22 23.02 long_context_needle prose 5494 216.48 22.22 tool_calling_format struct 111 80.12 23.89
Would be so kind to share the command you've run to obtain this? Thank you!
Spec- ryzen 9 5950x, 64GB ddr4, 2x RTX 3080 10GB (total 20GB vram)
offloaded n-gram to nvme ssd , and getting 16-17t/s on Q4_k_XL
What is your llama.cpp launch parameters?
Spec- ryzen 9 5950x, 64GB ddr4, 2x RTX 3080 10GB (total 20GB vram)
offloaded n-gram to nvme ssd , and getting 16-17t/s on Q4_k_XL
What is your llama.cpp launch parameters?
-m <path_to_model.gguf>
-c 262144
-n -1
-np 1
-t 16
-tb 16
-b 2048
-ub 512
--cache-type-k q8_0
--cache-type-v q8_0
--load-mode mmap
--flash-attn on
--jinja
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--host 127.0.0.1
--port 10000
Fable proposed me to post this here:
2× L40S (48GB, Ada SM89) + EPYC 160c + 750GB DDR5-12ch — Q8_0 hybrid, first Q8 numbers in this thread
Skeleton + KV + 8 expert layers on GPU (symmetric 4+4 split to balance the two cards),
40 expert layers + PLE/n-gram tables on CPU. 6 slots × 262k native ctx (unified KV pool).
llama-server
-m Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf --mmproj mmproj-BF16.gguf
-ngl all -sm layer -ts 1,1
-ot "per_layer_token_embd=CPU,ple_ngram_embd=CPU,blk.([4-9]|[1-3][0-9]|4[0-3]).ffn_.*_exps.=CPU"
--parallel 6 -c 1572864 -fa on -ctk q8_0 -ctv q8_0
-t 36 -tb 80 -b 8192 -ub 4096 --load-mode none --jinja --reasoning-preserve
Results (llama-server, real requests):
- pp: 447 t/s @ 2.5k → 679 t/s @ 68k (ubatch amortization: pp increases with prompt length)
- tg: 41 t/s fresh → 24 t/s @ 68k depth
- llama-bench depth curve (pp4096/tg64): 824/41.6 @ 0 → 609/24.9 @ 64k → 472/18.2 @ 128k → 360/12.8 @ 224k
- ub8192 + single slot reaches 954 t/s pp but OOMs with the 6-slot pool (compute buffers ~28.7 GiB/GPU)
- ctv f16 is a trap at depth on GPU: tg halves vs q8_0 (16.6 vs 30.3 @ 32k)
- CPU-only baseline (same box): 174 pp / 14.6 tg
Notes: -sm row not supported on this arch; mono vs dual GPU identical for single-stream
(pipeline costs nothing, second card = capacity). Watch --threads-batch: 160 threads
collapsed pp to 0.8 t/s, 80 is optimal on this EPYC.
4x RTX 5090, UD-Q4_K_XL, llama.cpp built at the merged PR head, 131,072 ctx:
| depth | decode | prefill |
|---|---|---|
| 4.5K | 84 tok/s | 1,510 tok/s |
| 23K | 62 | ~1,650 |
| 59K | 40 | ~1,650 |
| 93K | 32.5 | 1,583 |
The decode decay with depth others are seeing is real on llama.cpp today: existing GGUFs predate the MTP-head export, so there is no drafter to amortize the sparse-attention scan (the convert-side support is in PR #27836, GGUFs need re-exporting).
Same model and hardware on vLLM with the built-in MTP head holds 188-206 tok/s flat from 99K to 224K, no decay. Patched-nightly recipe with exact flags: https://github.com/abtraore/QWEN-PEDIA/tree/main/recipes/qwen3.8-flash-next
With my branch containing several graph (and metal specific) optimizations (https://github.com/tarruda/llama.cpp/tree/qwen4exp-metal-optimizations), this is what I get on a M1 ultra 128G:
% ./build/bin/llama-bench -m ~/qwen-3.8-next/Qwen3.8-Flash-Next-IQ4_NL.gguf -lm mmap -n 128 -p 512 -d 0,10000,20000,30000,40000,50000,60000,70000,80000,90000,100000,150000,200000
| model | size | params | backend | threads | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ---------: | --------------: | -------------------: |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 | 543.85 ± 6.90 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 | 39.48 ± 0.16 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d10000 | 465.45 ± 1.72 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d10000 | 35.28 ± 0.30 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d20000 | 449.96 ± 6.48 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d20000 | 35.50 ± 0.15 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d30000 | 439.84 ± 5.78 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d30000 | 34.70 ± 0.07 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d40000 | 427.61 ± 9.28 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d40000 | 34.44 ± 0.04 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d50000 | 296.36 ± 9.37 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d50000 | 29.78 ± 1.66 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d60000 | 408.97 ± 9.73 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d60000 | 33.15 ± 0.56 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d70000 | 398.93 ± 12.17 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d70000 | 32.87 ± 0.29 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d80000 | 391.81 ± 14.13 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d80000 | 32.60 ± 0.04 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d90000 | 383.34 ± 13.49 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d90000 | 32.04 ± 0.09 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d100000 | 373.78 ± 16.60 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d100000 | 31.64 ± 0.05 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d150000 | 339.39 ± 18.87 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d150000 | 29.55 ± 0.15 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | pp512 @ d200000 | 309.67 ± 22.75 |
| qwen4exp A3B Q8_0 | 95.20 GiB | 176.94 B | MTL,BLAS | 16 | mmap | tg128 @ d200000 | 28.16 ± 0.09 |
CPU: Intel Core i7-12700KF
Windows sees: 8 cores / 16 threads
GPU: NVIDIA GeForce RTX 5070 Ti 16 GB
VRAM reported: 16303 MiB
RAM: 128 GB
Driver: NVIDIA 610.88
OS: Windows
Model: unsloth/Qwen3.8-Flash-Next-GGUF
Quant: UD-Q4_K_XL
Files: split GGUF, 4 parts
Total model size: ~103.7 GiB
mmproj: mmproj-F16.gguf
Context tested: 131072
llama.cpp: b10715, CUDA 13.3 Windows build
Prompt tokens: 25229
Completion tokens: 96
Prefill: 627.98 tok/s
Decode: 14.89 tok/s
VRAM: 13392 MiB / 16303 MiB
Working Set RAM: ~74.34 GB
llama-server.exe ^
-m "C:\AI\models\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf" ^
--mmproj "C:\AI\models\Qwen3.8-Flash-Next-mmproj-F16.gguf" ^
-a qwen3.8-flash-next-ud-q4-k-xl ^
--host 127.0.0.1 ^
--port 8089 ^
--ctx-size 131072 ^
--n-gpu-layers 99 ^
--n-cpu-moe 48 ^
--override-tensor "per_layer_token_embd.weight=CPU" ^
--load-mode none ^
--flash-attn on ^
--cache-type-k q8_0 ^
--cache-type-v q8_0 ^
--batch-size 4096 ^
--ubatch-size 2048 ^
--threads 8 ^
--threads-batch 8 ^
--parallel 1 ^
--jinja ^
--image-min-tokens 1024 ^
--image-max-tokens 4000 ^
--reasoning off ^
--temp 0.7 ^
--top-p 0.80 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 1.5 ^
--repeat-penalty 1.0 ^
--metrics
CPU: Ryzen 9 9950X3D2
RAM: 128GB
GPU: 2 * RTX PRO 5000 Blackwell 72GB
Running early vLLM 0.29.0 nighly build.
FP8 quant fits perfectly in VRAM with single 262k context. VLLM reports max concurrency 1.00-1.01x. No space for MTP.
Prefill is difficult to measure because of vLLM cache prefixing. Typically I see something about 2000 to 8000 TPS and under 300-400ms TTFT.
Decode is rock solid 102-105 TPS independent of context size. It's real magic. I never have seen anything like this with any other model. Some n-gram magic I think.
I want to test NVFP4 quant soon. Paired with MTP it should be real magic.
Also I have second rig with dual Radeon R9700 and I haven't tested it yet. But it looks like that build is much better for Qwen3.8-27B-FP8 with vLLM 0.28.0+ because of AITER core support for RDNA (it works).
1. Model file: unsloth/Qwen3.8-Flash-Next-GGUF, UD-Q2_K_XL (73.4 GB, 3 shards) + mmproj-BF16, MTP disabled
2. GPU: NVIDIA RTX 5090 32 GB (driver 610.43.03, PCIe 5.0 x16, CUDA 13.3, built for sm_120a)
3. CPU: Intel Core i7-14700KF (8P + 12E, 28 threads)
4. RAM: 128 GB DDR5-5200
5. Generation speed: 47–62 t/s from 2k to 239k context (stock llama.cpp with the same offload: 32 → 12 t/s over the same range), table below
6. App / frontend: llama.cpp fork (llama-server behind llama-swap), Linux 7.1 (Arch)
7. Launch parameters:
llama-server -ngl 999 -fa on -ctk q4_0 -ctv q4_0 --jinja \
--model Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf --mmproj mmproj-BF16.gguf \
-ot per_layer_token_embd=CPU -ncmoe 37 -t 25 -ub 1024 -lm none -lzm off \
-c 262144 -np 1 --moe-expert-cache 120
8. Extra notes
Full 262144 context with vision, one request at a time. 37 of the 48 expert layers on the CPU, 11 in VRAM, the per-layer embedding table (PLE, 27 GB) resident in RAM. VRAM peak 30.5 GB of 32.6 (measured over load + 239k prefill + decode), RSS 62 GB, cold load 41 s.
Benchmarks: one request per row, 128 generated tokens, temperature 0, no prompt cache, page cache evicted before the load. "Stock" is llama.cpp master e70802a with the same offload (f16 KV, -ncmoe 36, no expert cache), the setup I ran before.
| prompt tokens | prefill t/s, stock → fork | decode t/s, stock → fork |
|---|---|---|
| 2 239 | 769 → 819 | 31.9 → 47.2 |
| 15 538 | 934 → 997 | 36.3 → 62.2 |
| 72 941 | 832 → 939 | 23.0 → 55.8 |
| 128 307 | 776 → 885 | 18.3 → 58.1 |
| 239 260 | 632 → 776 | 12.2 → 57.1 |
Decode is flat across the whole context instead of sliding to 12 t/s at 239k.
Perplexity (wikitext-2, -c 8192, 16 chunks): UD-Q2_K_XL 4.363 with f16 KV, 4.382 with q4_0 KV (+0.4%). For reference the NVFP4 quant of this model (routed experts NVFP4, everything else BF16, 168 GB with the PLE in BF16) scores 3.969 but decodes 25% slower and prefills 12% slower on this box: the Blackwell FP4 tensor cores only serve batched prefill, and the CPU-side experts pay for the 45% extra bytes.
What was changed. A llama.cpp fork on master 159b74142 (b10767), everything behind env switches and A/B'd on one binary:
- Sparse attention decode path (qwen4exp QSA). The 4-head lightning indexer has no CUDA kernel in master (64/32 heads only), so it ran on the CPU with 16 MB of device-to-host copies per layer per token; I added the 4-head kernel. Top-k moved from tokens to the compressed blocks (CUB), the selected blocks are gathered and attended with the flash-attention vector kernel instead of masking the whole cache (from 64k cells up; below that masking is cheaper), the pooled indexer keys are cached in VRAM with only dirty blocks recomputed, the indexer KV cache stores K only (−1.6 GB VRAM at 262k), and the host-side block tables and the KQ mask are updated incrementally instead of rescanning every cell (5 ms → 0.5 ms of host time per token at 262k). GPU time per token at 128k went from 16.7 ms to 7.9 ms, the same as at 0 context. The gather idea was found independently in the same days by #27977/#28244 and #28213, and the K-only indexer cache by #28330; none merged yet.
- Hyper-connection fusion. The residual combine and the stream mean go through the DeepSeek-V4
dsv4_hc_post/dsv4_hc_prekernels (193 fused nodes per token), bitwise identical output, +2–4%. - Expert LRU cache (PR #27861 by csantiago78). GPU-resident slots for the host experts,
--moe-expert-cache N. This is the big lever together with a quantized KV cache: q4_0 KV frees 4.8 GB at 262k, spent on 120 slots per host layer, +25% decode over f16 KV without slots. Trading whole host layers for slots at equal VRAM is flat, so 37 layers stay on the host for a 1.5 GB VRAM margin. -lzm off. Master since #27837 (Aug 30) mmaps tensors marked lazy — the 27 GB PLE — even with-lm none. Every prefill token then faults 48 rows in from the page cache or the disk; here that halves prefill (pp16384: 508 vs 1365 t/s) and adds 50 s to the load, and it is the reason my numbers looked bimodal from run to run.--lazy-mode offkeeps the table resident. Same regression as #28160 (Vulkan iGPU) and #28355 (Windows), fix in progress in #28326; #28136 takes another route (direct reads) for GB10.
Things that did not help, for the record: re-encoding the host experts from IQ2_XS to Q4_0 (−12..−30%, the CPU path is byte-bound), -t 8/16/28 or P-core pinning (−10..−25%), --poll 100 (neutral), -ub 2048 (+28% prefill at 8k, OOM at 262k), more cache inserts per step (flat), --spec-type ngram-mod (+16% when the answer quotes the prompt, −1..−2% on prose, left as an option), MTP drafting (not tried here: the verification batch runs 3–4× the experts on the CPU; #28104 reports +50% at 70k on Vulkan).
All comparisons are interleaved A/B on one binary; this machine has a ±10% decode spread from CPU clocks and foreign load, so each number above is a single run.