unsloth

/

Qwen3.8-Flash-Next-GGUF

Like 982

Follow

Unsloth AI 35.5k

ok but i dont have a god gpu

but i could run this on colab!

When MTP support will be merged to llama.cpp upstream?

ok but i dont have a god gpu

You dont need to

When MTP support will be merged to llama.cpp upstream?

We're working on it

Hi, thank you for all your work. Can you advise if the mtp feature offloads to metal when it can't fit in vram? (256gb metal, 56gb gpu)

llama.cpp/llama-server
-hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
-md unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf
--spec-type draft-mtp --spec-draft-n-max 2

I cmn init: llama threadpool init, n_threads = 64
I common_speculative_init_result: loading draft model 'unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2647.04 MiB on device 1: cudaMalloc failed: out of memory
E alloc_tensor_range: failed to allocate CUDA1 buffer of size 2775623424
E llama_model_load: error loading model: unable to allocate CUDA1 buffer
E llama_model_load_from_file_impl: failed to load model
E common_speculative_init_result: failed to load draft model, 'unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
E srv load_model: failed to load draft model, 'unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
srv operator(): operator(): cleaning up before exit...
E srv llama_server: exiting due to model loading error

Seeing the same with smaller -shared-Q4 as well.

Hi, thank you for all your work. Can you advise if the mtp feature offloads to metal when it can't fit in vram? (256gb metal, 56gb gpu)

@modelmode . with your lauch setting llama.ccp is reserving space in vram for context multipied by parralele slots (default 4), so you eather optimize setting a limited context or/and limit parallel slots. you must find yout balance.
or run llama-serve with flag -ngld 0 to force load MTP to system RAM

weirdly I don't have any settings icons for models loaded by Unsloth Desktop itself, but it shows the settings for models from Lm studio:

This comment has been hidden

This reply has been hidden

hi, I build the llama-server as said in https://unsloth.ai/docs/models/qwen3.8-next#mtp-llama.cpp-guide, but report the following error:

0.00.597.788 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.01.096.341 I srv    load_model: loading model 'llama/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.01.297.515 E llama_model_load: error loading model: unknown model architecture: 'qwen4exp'
0.01.297.528 E llama_model_load_from_file_impl: failed to load model
0.01.297.563 W srv    load_model: [spec] failed to measure draft model memory: failed to load model
0.01.497.361 E llama_model_load: error loading model: unknown model architecture: 'qwen4exp'
0.01.497.366 E llama_model_load_from_file_impl: failed to load model
0.01.497.379 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.01.694.125 E llama_model_load: error loading model: unknown model architecture: 'qwen4exp'
0.01.694.129 E llama_model_load_from_file_impl: failed to load model
0.01.694.133 E cmn  common_init_: failed to load model '/llama/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.01.694.134 E srv    load_model: failed to load model, '/llama/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.01.694.138 I srv    operator(): operator(): cleaning up before exit...
0.01.701.068 E srv  llama_server: exiting due to model loading error

sorry, I enter the wrong llama.cpp path

x3 3090 192GB RAM 6250R x2 XEON
Why it does not load?

Gemini and ChatGPT cant fix it: (

K:\llamacpp4>cd /d K:\llamacpp4

K:\llamacpp4>llama-server.exe -m "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf" -md "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" --alias "Qwen3.8-Flash-Next-Q4-XL-MTP" --mmproj "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mmproj-F16.gguf" -sm layer -fit on -fitt 1536,1536,1536 -c 131072 -np 1 -fa on -b 2048 -ub 1024 -t 8 -ctk q8_0 -ctv q8_0 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --top-k 20 --top-p 0.95 --presence-penalty 0 --repeat-penalty 1.0 --port 8080 --chat-template-kwargs {"preserve_thinking":true} --reasoning-format deepseek --image-min-tokens 1024 --seed 9000
0.00.247.808 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.248.355 W srv llama_server: -----------------
0.00.248.356 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.248.356 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.248.357 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.248.357 W srv llama_server: -----------------
0.00.258.652 I srv load_model: loading model 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf'
0.02.124.307 E llama_model_load: error loading model: borrow_shared_tensor: this model is a draft head without its own 'token_embd.weight'; load it as a draft of its target model, not on its own
0.02.124.319 E llama_model_load_from_file_impl: failed to load model
0.02.124.397 W operator (): failed to measure the memory of the extra model, fitting without it: failed to load model
0.18.081.288 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance
1.09.330.692 I cmn init: llama threadpool init, n_threads = 8
1.10.107.657 I common_speculative_init_result: loading draft model 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
1.11.293.653 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2647.04 MiB on device 2: cudaMalloc failed: out of memory
1.11.293.663 E alloc_tensor_range: failed to allocate CUDA2 buffer of size 2775623424
1.11.374.500 E llama_model_load: error loading model: unable to allocate CUDA2 buffer
1.11.374.740 E llama_model_load_from_file_impl: failed to load model
1.11.374.742 E common_speculative_init_result: failed to load draft model, 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
1.11.374.762 E srv load_model: failed to load draft model, 'D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf'
1.11.374.773 I srv operator (): operator (): cleaning up before exit...
1.11.391.101 E srv llama_server: exiting due to model loading error

K:\llamacpp4>pause
Press any key to continue . . .

cd /d K:\llamacpp4

llama-server.exe ^
-m "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf" ^
-md "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" ^
--alias "Qwen3.8-Flash-Next-Q4-XL-MTP" ^
--mmproj "D:\Downloads\LLM models\LMStudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\mmproj-F16.gguf" ^
-sm layer ^
-fit on ^
-fitt 1536,1536,1536 ^
-c 131072 ^
-np 1 ^
-fa on ^
-b 2048 ^
-ub 1024 ^
-t 8 ^
-ctk q8_0 ^
-ctv q8_0 ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
--spec-draft-n-min 0 ^
--top-k 20 ^
--top-p 0.95 ^
--presence-penalty 0 ^
--repeat-penalty 1.0 ^
--port 8080 ^
--chat-template-kwargs {"preserve_thinking":true} ^
--reasoning-format deepseek ^
--image-min-tokens 1024 ^
--seed 9000

pause

x3 3090 192GB RAM 6250R x2 XEON
Why it does not load?

Gemini and ChatGPT cant fix it: (

...

allocating 2647.04 MiB on device 2: cudaMalloc failed: out of memory

For whatever reason it didn't fit on your GPU(s).

multi gpu and buffer size got fixed please check this pr

export LLAMA_PLE_RESIDENT=1
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp/build/bin/llama-server
--model /mnt/nvme/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf
--chat-template-file /mnt/nvme/or.jinja
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'
--model-draft /mnt/nvme/mtp-Qwen3.8-Flash-Next-BF16.gguf
-devd CUDA3
--spec-type draft-mtp
--spec-draft-n-max 3
--tensor-split 3.2,4,1.7,0
-cmoe
-cmoed
--numa distribute
-c 260000
--batch-size 7100
--ubatch-size 7100
--load-mode none
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--parallel 1
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja

https://github.com/ggml-org/llama.cpp/pull/28623

the above is example so use it one your test the ple is put the ngram in ram and also load non also do the same so test it properly with your flags and share your commend

i spend a lots of time for that i hope its helps some people

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

@techhermit same here, it was working before b11007-mix-3e83366 llama.cpp update

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

@techhermit same here, it was working before b11007-mix-3e83366 llama.cpp update

I am still on b10837

i cannot use IQ3_XXS with the MTP Q4 either token_embd.weight not found for the shared or output_hc_norm.weight not found for the non-shared

@techhermit same here, it was working before b11007-mix-3e83366 llama.cpp update

I am still on b10837

check if it is forcing the use of mtp-Qwen3.8-Flash-Next-Q8_0.gguf or mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf
was working for me with the shared one untill last updates (bundled llama + unsloth)