havenoammo

/

Qwen3.6-35B-A3B-MTP-GGUF

Like 84

Hmm. Getting this error trying to run the MTP model.

load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
llama_model_load: error loading model: vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
llama_model_load_from_file_impl: failed to load model
srv load_model: failed to load MTP head from 'Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf'
srv operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

Also, I tried a normal model and the normal models seems about 3 times slower now. Only getting like 2.1 tokens/second on the regular models when I was getting 7-11 tokens/second before merging.

If regular models got slower, CUDA may not have been present during the build, or you might be on non-Nvidia hardware. Let me know what hardware you have and I'll try to help!
The model load issue might be a corrupted download. Can you check the file hash? On Linux, run sha256sum <filename> and it should print ab94e2da12d2bdc22777ba1b7422bbf8d5d9d0bee1164ca7343a0cee3310038a.

I am building with CUDA 12.8 (Was issue with 13.1 and had to add the 12.8 toolkit - would compile but seg faulted no matter what you tried.)

Compiling again to see if it makes any difference. My branch was showing main/master so trying again in case the merge failed. But if it did not change, does not explain slowdown running normal model.

It says [ 21%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-iq2_xs.cu.o

And I have MSI Afterburner up to see if GPU is being used. It was running, but the usage was low.. like running 800mhz instead of 2050mhz

jeffc@Crypto2:~$ sha256sum Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf
ab94e2da12d2bdc22777ba1b7422bbf8d5d9d0bee1164ca7343a0cee3310038a Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf

Looks good

jeffc@Crypto2:~/llama.cpp$ git status
On branch pr-22673

Still getting same error:

load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
llama_model_load: error loading model: vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
llama_model_load_from_file_impl: failed to load model
srv load_model: failed to load MTP head from 'Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf'
srv operator(): operator(): cleaning up before exit...
main: exiting due to model loading error
jeffc@Crypto2:~$

It's weird because it loads all the way up to the end. It like fails after it finishes loading. But the whole loading process looks good up to that message.

I just noticed.. when I run a regular model...

llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 2070) (0000:02:00.0) - 7144 MiB free

When I try running the MTP model:

llama_model_create: overriding architecture qwen35moe -> qwen35moe_mtp
llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 2070) (0000:02:00.0) - 0 MiB free

Notice how the MiB free is 0 ... That is the issue.

I tried removing all the optional parameters and it still fails so there is something seriously wrong with the MTP code.

I ran the diff changes through DeepSeek and came up with some code changes. Four code fixes. Compiling now. See if the fixes actually fix the issue. Nope. Didn't fix it.

Modifications to:
src/llama-model-loader.cpp
src/llama-model.cpp
src/llama-memory-recurrent.cpp
tools/server/server-context.cpp

Did a git clone https://github.com/am17an/llama.cpp.git
To grab right from your version. Compiling now with everything set to Cuda 12.8.
See if that works. NOPE. But the 0MiB reading for the VRAM was not causing the issue. This version the VRAM is displayed correctly.
But still getting the same error (This part is new: llama_model_load: error loading model: missing tensor 'blk.40.ssm_conv1d.weight') trying to run a MTP model:

load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
llama_model_load: error loading model: missing tensor 'blk.40.ssm_conv1d.weight'
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf'
srv load_model: failed to load model, 'Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf'
srv operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

That seems odd. Later patches to the main branch after I made these models may have caused this, or VRAM allocation in the PR might not be fully tested yet. Try this first:

git reset --hard 5207d120e
git merge --no-ff pr-22673 -m "Merge PR #22673: llama + spec: MTP Support"
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release --target llama-server

If that doesn't work, try this one, which is the exact commit I tested and am currently using daily:

git reset --hard e3e3f8e46
git merge --no-ff pr-22673 -m "Merge PR #22673: llama + spec: MTP Support"
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release --target llama-server

I also tested the MoE model with MTP and everything worked on my end. One thing to note is that MTP layers take around 903MB extra VRAM, and it's possible this isn't counted during initialization, causing a failure afterwards. I'm not the author of the PR so they might be able to provide better insight and a fix.

If VRAM is the issue, there's an argument for offloading MoE layers to CPU which might help. Can you share your full run command? Maybe someone else here can help too.

I tried the master branch, the mtp-clean, and the mtp-on-22679. They all have the same error. Those commands you pasted don't work. Can you verify this is from https://github.com/am17an/llama.cpp
Trying a fresh git clone again.

jeffc@Crypto2:~/llama.cpp$ git reset --hard e3e3f8e46
fatal: ambiguous argument 'e3e3f8e46': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git [...] -- [...]'

Command I am running (or trying to. lol)

llama-server --model "Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf" --port 8001 --host 0.0.0.0 --ctx-size 131072 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence_penalty 1.5 --repeat_penalty 1.0 --threads 6 --threads-batch 8 --jinja --reasoning off -ctk q4_0 -ctv q4_0 --n-cpu-moe 35 --no-mmap --batch-size 2048 --ubatch-size 2048 --spec-type mtp --spec-draft-n-max 3

Hmm you are on wrong repo it seems I was using the official repo, can you try cloning and building this one: https://github.com/ggml-org/llama.cpp

Having issues running this with ik-llama. 28B worked fine.

llama_model_load: error loading model: check_tensor_dims: tensor 'blk.40.attn_qkv.weight' not found
llama_model_load_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'C:\Users\furka.lmstudio\models\qwen\Qwen3.6-35B-A3B-UD-Q5_K_XL-MTP.gguf'
ERR [ load_model] unable to load model | tid="26612" timestamp=1778193564 model="C:\Users\furka\.lmstudio\models\qwen\Qwen3.6-35B-A3B-UD-Q5_K_XL-MTP.gguf"

Having issues running this with ik-llama. 28B worked fine.

llama_model_load: error loading model: check_tensor_dims: tensor 'blk.40.attn_qkv.weight' not found
llama_model_load_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'C:\Users\furka.lmstudio\models\qwen\Qwen3.6-35B-A3B-UD-Q5_K_XL-MTP.gguf'
ERR [ load_model] unable to load model | tid="26612" timestamp=1778193564 model="C:\Users\furka\.lmstudio\models\qwen\Qwen3.6-35B-A3B-UD-Q5_K_XL-MTP.gguf"

Not sure about ikllama's MTP implementation, they might be missing the qwen35moe architecture support or something similar. I just tested Q5 on my llama.cpp setup and it worked fine.
draft acceptance rate = 0.59348 ( 892 accepted / 1503 generated)
SHA256 of the model: 45b5f79aadfe4da33bb8082d000efa677aa81422e1d9fcf03f9037f6743e7af7

Hmm you are on wrong repo it seems I was using the official repo, can you try cloning and building this one: https://github.com/ggml-org/llama.cpp

I tried from both. Tried 3 different MTP models as well. Tried a ton of different merges and everything ends in that stupid error.
Guess I will have to wait for the official release of this feature.

load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
llama_model_load: error loading model: vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
llama_model_load_from_file_impl: failed to load model
srv load_model: failed to load MTP head from 'Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf'
srv operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

Having issues running this with ik-llama. 28B worked fine.

llama_model_load: error loading model: check_tensor_dims: tensor 'blk.40.attn_qkv.weight' not found
llama_model_load_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'C:\Users\furka.lmstudio\models\qwen\Qwen3.6-35B-A3B-UD-Q5_K_XL-MTP.gguf'
ERR [ load_model] unable to load model | tid="26612" timestamp=1778193564 model="C:\Users\furka\.lmstudio\models\qwen\Qwen3.6-35B-A3B-UD-Q5_K_XL-MTP.gguf"

Not sure about ikllama's MTP implementation, they might be missing the qwen35moe architecture support or something similar. I just tested Q5 on my llama.cpp setup and it worked fine.
draft acceptance rate = 0.59348 ( 892 accepted / 1503 generated)
SHA256 of the model: 45b5f79aadfe4da33bb8082d000efa677aa81422e1d9fcf03f9037f6743e7af7

Just wanted to give you a heads up on this. I tried every fork and branch mentioned here and nothing would work on my 8 gig 3060 ti (cuda12.9) until I caught your suggestion about CPU-MOE. I tried 31 at first to be sure and BOOM, model loaded. Then I tried 1 layer and BOOM loaded. Looks like when the MTP layer is on GPU it wont work but CPU it will.

Also, thanks so much for all your hard work on this! Now its working im getting around 68-75% acceptence rate with a boost of about 25-35% speed increase FOR ABSOLUTELY NOTHING! Honestly, what a time to be alive lol. My lowly 8 gig card can run qwen 3.6 35ba3b at speeds I never imagined!