peculiar-ragdoll

/

Tiel-Coder-35B-A3B-GGUF-MTP

like 16

Model: peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF-MTP, quant UD-Q4_K_XL (24 GB file)
Runtime: LM Studio, engine llama.cpp-linux-x86_64-nvidia-cuda-avx2@2.29.1
GPU: Tesla V100-PCIE-32GB (SM 7.0), driver 580.178.04, Linux
Config: 262144 context (also reproduced at 131072), KV cache F16 (also reproduced at K=Q8/V=Q8), flash attention default

Symptom

With MTP enabled, VRAM grows continuously while the model runs and
never plateaus: 22 GB after weight load → ~27 GB after KV/context
allocation → climbs steadily during prompt processing / generation of a
long request (50–80k-token prompts) until the card is full, then the
inference worker aborts and LM Studio reloads the model:

E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 648.03 MiB on device 0: cudaMalloc failed: out of memory
...
E CUDA error: CUBLAS_STATUS_INVALID_VALUE
  llm-engine/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:106: CUDA error
  ggml_abort <- ggml_cuda_error <- ggml_cuda_mul_mat_cublas
  <- ggml_backend_cuda_graph_compute <- llama_context::process_ubatch
  <- llama_context::decode  (server update_slots loop)
[Inferior 1 (process N) detached]
[ERROR][tiel-coder-35b-a3b-mtp] Error: Channel Error

The CUBLAS_STATUS_INVALID_VALUE appears to be downstream fallout of a
failed pool allocation (a null/invalid workspace handed to cuBLAS), not
an independent type bug. One crash also logged a single implausible
21,656 MiB buffer request.

Control experiment (isolates MTP)

Same GGUF, same machine, same settings, MTP disabled: VRAM is
completely static at 26.8 GB — sampled every 20 s across many minutes
while generating 15k+ tokens over a 262k context. Identical
static-after-load behavior to non-MTP sibling models (Ornith-1.0-35B,
Ornith-1.5-35B-A3B) which run the same workloads on this card without
any growth.

Conclusion: the growth is specific to the MTP path — consistent with
per-position MTP structures (e.g. cached hidden states feeding the MTP
head) being allocated from the on-demand CUDA pool instead of being
included in the load-time worst-case reservation, and growing with
tokens processed until allocation fails mid-graph.

Repro

1. Load the model with MTP enabled on a 32 GB card, long context.
2. Send a single long prompt (≥50k tokens) and let it generate.
3. Watch nvidia-smi: memory climbs monotonically until the abort.
4. Repeat with MTP disabled: memory never moves after load.