From: opencoti
Subject: [PATCH 0090] multi-session graph-node budget for --parallel >=2 (bug-2094)

graph_max_nodes() sized the compute-graph meta pool as a single-stream budget
(8*n_tensors for dense archs; max(n_tokens*40, 32*n_tensors) for the qwen3next/
qwen35/kimi linear-attn branch), INDEPENDENT of the stream count. But with a
non-unified KV cache (the default; --parallel N => kv_unified=false) the graph
builds per-stream KV views/masks/copies (the `for s < n_stream` loops throughout
llama-kv-cache.cpp), so the actual node count scales with the number of streams.
On a 48-layer model (Qwen2.5-14B-Instruct-1M) --parallel 2 therefore overflowed
the pool at graph build:

    llama.cpp/ggml/src/ggml.c:1790: not enough space in the context's memory pool

(server aborts at boot; --parallel 1 was fine; independent of --kv-residency-mode
and of ctx). Fix: scale the node budget by n_stream = (kv_unified ? 1 : n_seq_max),
applied to both branches. Tied to the exact per-stream-KV-graph mechanism; a
UNIFIED cache keeps n_stream=1 -> byte-identical to upstream, so the factor
self-disables when there is no per-stream graph.

Measured actual n_nodes (RTX 3090, ctx 8192): dense Qwen3-8B 1267/2131/3571 at
--parallel 1/2/4 (~776 nodes/stream); iSWA Gemma-4-A4B 2647/3367/4567 (~660/stream);
qwen35 Qwen3.5-9B 1856/2048/2368 (~170/stream). All boot with 2.5-4x headroom.
Verified on the original repro (bs2 RTX PRO 6000, 14B-1M-Q8_0 --parallel 2): CRASH
-> BOOT. Host-only (build:llamafile:make); no CUDA DSO, no restamp.

--- a/llama.cpp/src/llama-context.cpp
+++ b/llama.cpp/src/llama-context.cpp
@@ -2585,10 +2585,19 @@
 //
 
 uint32_t llama_context::graph_max_nodes(uint32_t n_tokens) const {
+    // opencoti-hook: bug-2094 multi-session graph-node budget — see docs/features/rolling_kv_compute_reserve.md
+    // With a non-unified KV cache (--parallel N ⇒ kv_unified=false) the graph builds
+    // per-stream KV views/masks/copies (the `for s < n_stream` loops throughout
+    // llama-kv-cache.cpp), so the node count scales with the stream count. The upstream
+    // budgets below are single-stream; without this factor --parallel ≥2 overflows the
+    // graph-meta pool (ggml.c:1790 "not enough space in the context's memory pool"). A
+    // UNIFIED cache keeps n_stream=1 → byte-identical to upstream, so the factor
+    // self-disables exactly when there is no per-stream graph.
+    const uint32_t n_stream = cparams.kv_unified ? 1u : std::max<uint32_t>(1u, cparams.n_seq_max);
     if (model.arch == LLM_ARCH_QWEN3NEXT || model.arch == LLM_ARCH_KIMI_LINEAR || model.arch == LLM_ARCH_QWEN35 || model.arch == LLM_ARCH_QWEN35MOE) {
-        return std::max<uint32_t>(n_tokens * 40, 32u * model.n_tensors());
+        return std::max<uint32_t>(n_tokens * 40, 32u * model.n_tensors() * n_stream);
     }
-    uint32_t res = std::max<uint32_t>(1024u, 8u*model.n_tensors());
+    uint32_t res = std::max<uint32_t>(1024u, 8u*model.n_tensors()) * n_stream;
     for (const auto & lora : model.loras) {
         res += lora->get_n_nodes();
     }