MTP draft model fails to load in llama.cpp — "invalid vector subscript" (gemma4-assistant arch)
Hi, thanks for the great GGUFs! I'm hitting a consistent crash when trying to use the MTP draft model for speculative
decoding. The main model loads and runs perfectly, but llama-server crashes while loading the MTP draft, regardless of
build version or MTP quant.
Error (identical every time):
srv load_model: loading draft model '...gemma-4-12B-it-MTP-Q8_0.gguf'
llama_model_load: error loading model: invalid vector subscript
llama_model_load_from_file_impl: failed to load model
srv load_model: failed to load draft model
srv llama_server: exiting due to model loading error
What I tried (all crash with the same invalid vector subscript):
┌────────────────────────────┬──────────┬────────┐
│ llama.cpp build │ MTP file │ Result │
├────────────────────────────┼──────────┼────────┤
│ b9702 (CUDA 13.3, Win x64) │ MTP-Q8_0 │ crash │
├────────────────────────────┼──────────┼────────┤
│ b9717 (CUDA 13.3, Win x64) │ MTP-Q8_0 │ crash │
├────────────────────────────┼──────────┼────────┤
│ b9717 (CUDA 13.3, Win x64) │ MTP-BF16 │ crash │
└────────────────────────────┴──────────┴────────┘
So it's not a quantization issue and not a stale-build issue — the loader fails on the gemma4-assistant architecture
itself.
Command:
llama-server.exe -m gemma4-v2-Q8_0.gguf ^
--spec-type draft-mtp ^
--spec-draft-model gemma-4-12B-it-MTP-Q8_0.gguf ^
--spec-draft-ngl 99 --ctx-size 65536 -ngl 99 --flash-attn on --jinja
Earlier in the load there's also this (you noted it's "normal during memory fitting", just including for
completeness):
llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set
srv load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
Draft model GGUF metadata (parsed from the file):
- general.architecture = gemma4-assistant
- block_count = 4, nextn_predict_layers = 4
- embedding_length = 1024, embedding_length_out = 3840
- 49 tensors, 48 KV entries — header looks valid and complete (file size matches general.size_label = 423M).
Main model is general.architecture = gemma4, embedding_length = 3840, block_count = 48.
Environment: Windows 11, RTX 4070 Ti SUPER (16 GB), official llama.cpp Windows CUDA release builds.
Question: Was the MTP head built against a specific llama.cpp PR/fork? If so, could you note the exact build/commit in
the README? Or is the gemma4-assistant loader path possibly indexing a vector out of bounds for this config?
I provide full --verbose logs. Thanks!
https://gist.github.com/Metodv1/e6e06c9058fdd0b32881699e4ae00e24
Hi Metod, this is an excellent report — the metadata dump and gist made it easy to pin down, thank you. Short version:
the files are fine, it's a build regression, and I just reproduced the working case on my end.
To your direct question first: the MTP head is not built against any private fork. It's the unsloth-converted GGUF of
Google's official gemma-4-12B-it-assistant draft, and Gemma 4 MTP support is in llama.cpp mainline (it landed via PR
#23398, merged 2026-06-07). So no special fork — it just needs a build where that loader path is intact.
Here's the key finding: I ran your exact setup — gemma4-v2-Q8_0 as the main model + gemma-4-12B-it-MTP-Q8_0 as the
draft — on build b9553 (commit 9e3b928fd), and it loads cleanly with no "invalid vector subscript" and gives a real
speedup: ~88 tok/s without the draft vs ~180 tok/s with it on a simple deterministic prompt (expect more like 1.2–1.3×
on real coding/thinking, since this is the generic Gemma 4 draft, not retrained for v2 — still lossless).
So your instinct is right: the gemma4-assistant loader is going out of bounds, but it's build-specific. It works on
b9553 and crashes on b9702/b9717 — something in that loader/context-init path regressed between those builds. The
"Gemma4Assistant requires ctx_other to be set (normal during memory fitting)" line shows up on b9553 too and is
harmless; the fatal "invalid vector subscript" is the regression.
What to do:
- Grab the b9553 CUDA Windows release specifically. I'll pin that exact build/commit in the README so people aren't
guessing. - Heads-up on flags: b9553 uses the older names — use --model-draft (not --spec-draft-model), --spec-draft-n-max 4,
and -ngld 99 (not --spec-draft-ngl). So:
llama-server -m gemma4-v2-Q8_0.gguf --model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp
--spec-draft-n-max 4 -ngl 99 -ngld 99 -fa on --jinja - And honestly, please do file your gist as an upstream llama.cpp issue — it's a clean, well-documented
gemma4-assistant loader regression, and getting it fixed restores MTP on the newer builds for everyone (I'd rather it
work on latest than pin an old build forever).
Hi, it works perfectly — thank you so much for the detailed help! 🙏
I grabbed the b9553 CUDA 13.3 Windows build exactly as you suggested, used the older flag names (--model-draft,
--spec-draft-n-max 4, -ngld 99), and the MTP draft loads cleanly — no more invalid vector subscript. Your diagnosis
was spot on: it's a build regression, working on b9553 and broken on b9702/b9717.
Here are my numbers (RTX 4070 Ti SUPER, gemma4-v2-Q8_0 + MTP-Q8_0, deterministic coding prompt, temp 0):
Metric Without MTP With MTP
------------------ ----------- --------------
Generation speed ~41 tok/s ~101 tok/s
Speedup 1x ~2.5x
Prefill n/a ~130 tok/s
Draft acceptance n/a 181/293 (~62%)
Output quality baseline identical
Completely lossless, and a really nice boost on code. 🚀
As you suggested, I filed it as an upstream llama.cpp issue so it can get fixed on the latest builds for everyone:
👉 https://github.com/ggml-org/llama.cpp/issues/24795
Thanks again for the great models and for taking the time to reproduce it on your end — pinning b9553 in the README
will definitely save other people the headache. Cheers!
b9716 - I have no issues running it.
Windows 11.
For MTP I use this params: --spec-type draft-mtp --spec-draft-n-max 4 --model-draft "C:[AI]\models\gemma-4-12B-it-MTP-Q8_0.gguf"
~120 tok/s on 4090
It works for me without any issues.
incognito@gmktec-evo-x2:/mnt/data/dev/llama.cpp-vulkan-10.06.2026$ ./build/bin/llama-server -m /mnt/data/models/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/gemma4-v2-Q8_0.gguf --model-draft /mnt/data/models/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/MTP/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072 --n-gpu-layers 99 --n-gpu-layers-draft 99 --no-mmap -fa on --temp 1.0 --top-p 0.95 --top-k 64 --repeat-penalty 1.1 -np 1 --host 0.0.0.0 --port 18080
0.00.034.929 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.034.932 I device_info:
0.00.035.043 I - Vulkan0 : AMD Radeon Graphics (RADV GFX1151) (128000 MiB, 127557 MiB free)
0.00.035.047 I - CPU : AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (127432 MiB, 127432 MiB free)
0.00.035.096 I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.067.193 I srv init: running without SSL
0.00.068.440 I srv init: using 31 threads for HTTP server
0.00.069.131 I srv start: binding port with default address family
0.00.070.699 I srv llama_server: loading model
0.00.070.713 I srv load_model: loading model '/mnt/data/models/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/gemma4-v2-Q8_0.gguf'
0.00.527.791 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
0.00.614.955 W srv load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
0.00.614.990 I common_init_result: fitting params to device memory ...
0.00.614.991 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.293.102 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.293.288 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.302.898 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.02.766.193 W llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.02.890.836 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.03.404.205 I srv load_model: loading draft model '/mnt/data/models/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/MTP/gemma-4-12B-it-MTP-Q8_0.gguf'
0.03.675.043 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.03.675.223 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.03.684.398 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.03.787.886 W llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.03.788.394 W llama_kv_cache: layer 3: sharing with layer 47. k = 0x30001000, v = 0x38001000
0.03.788.412 W llama_kv_cache: layer 0: sharing with layer 46. k = 0x1d401000, v = 0x1da01000
0.03.788.414 W llama_kv_cache: layer 1: sharing with layer 46. k = 0x1d401000, v = 0x1da01000
0.03.788.414 W llama_kv_cache: layer 2: sharing with layer 46. k = 0x1d401000, v = 0x1da01000