From: opencoti Subject: [PATCH 5/N] lazy slot-context allocation — Phase 3 (shrink on idle) Phase 2 added a soft cap on per-stream KV cells (kv_size_alloc <= kv_size_max) and grow-on-demand via ensure_capacity. The soft cap never shrinks, so once a single long prompt has caused it to grow, the embedder's RSS stays at the grown size for the lifetime of the process. For an embedder running `--parallel 2 --ctx-size 16384` with typical prompts well under 4K cells, this means the headline Phase-2 win evaporates after the first long-context request. Phase 3 plugs that hole. When an idle threshold has elapsed since the last apply_ubatch: 1. Cells in the range [kv_size_shrink_target, kv_size_alloc) are evicted across every stream via cells.rm(i). The server-context layer's prompt cache keeps cells alive between embed requests for reuse; decommitting their backing pages without first evicting would leave the prompt cache pointing at zeroed memory and corrupt subsequent re-uses. The eviction makes the shrink unconditional. 2. The pages backing cells [target, current_alloc) of every layer's K and V tensors are returned to the kernel via madvise(addr, len, MADV_DONTNEED). 3. n_cells_cleared is rolled back to the shrink target so the next ensure_cleared() partial-memset re-zeros only what comes back. 4. kv_size_alloc is reset to the shrink target so the next request can grow again from the initial cap. Why madvise(MADV_DONTNEED) and not posix_madvise(POSIX_MADV_DONTNEED): glibc on Linux maps posix_madvise(POSIX_MADV_DONTNEED) to a no-op (see glibc sysdeps/posix/posix_madvise.c — the standard explicitly forbids "actively destroying" pages, which is what we need). Only the Linux- specific madvise(MADV_DONTNEED) actually decommits anonymous pages and returns RSS to the kernel. We declared MADV_DONTNEED with the canonical Linux value (4) under #ifndef and extern-declared madvise() so cosmocc's default feature set (which does not define _GNU_SOURCE and so does not expose madvise() through sys/mman.h) builds cleanly. The shrink check runs at the top of init_batch() and prepare() — both entry points to actual KV work. It is gated by: - slot_shrink_idle_ms_ > 0 (feature enabled) - kv_size_alloc > kv_size_shrink_target (something to shrink) - last_active_ms_ != 0 AND (now - last_active_ms_) >= idle threshold Surface (CLI): --slot-shrink-idle-ms N (default 0 = disabled; positive enables) env LLAMA_ARG_SLOT_SHRINK_IDLE_MS Surface (C API): llama_context_params::slot_shrink_idle_ms (uint32_t, default 0) Plumbed through llama_cparams -> llama_kv_cache ctor -> (llama_kv_cache_iswa ctor for SWA models) -> llama-model.cpp call sites. Backward-compat: llama-memory-hybrid{,-iswa}.cpp pass 0 (no shrink), the same way they already pass 0 for kv_size_initial. Bench (CPU, --gpu disable so KV lives in host pages): the embedder configured `--parallel 2 --ctx-size 16384 --pooling last --slot-initial-ctx 256 --slot-shrink-idle-ms 30000` starts at 719 MB RSS, grows to 1720 MB after a long-prompt corpus pass (soft cap 256 -> 2048), and after the 30 s idle drops to 1329 MB on the next short embed. 391.7 MB returned to the kernel — measured by /proc//status VmRSS. Quality (cosine min vs M2 baseline) stays at 0.999797 across the 48-prompt corpus. Milestone: F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md Upstreaming: TBD — kv_cache idle-shrink is a generally-useful concept; the cell-eviction + madvise(MADV_DONTNEED) approach is not opencoti-specific. Proposal candidate after F4 closes. --- diff --git a/llama.cpp/common/arg.cpp b/llama.cpp/common/arg.cpp --- a/llama.cpp/common/arg.cpp +++ b/llama.cpp/common/arg.cpp @@ -1371,6 +1371,17 @@ common_params_context common_params_parser_init(common_params & params, llama_ex params.slot_initial_ctx = value; } ).set_env("LLAMA_ARG_SLOT_INITIAL_CTX").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_EMBEDDING, LLAMA_EXAMPLE_PARALLEL})); + add_opt(common_arg( + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + {"--slot-shrink-idle-ms"}, "N", + string_format("idle threshold (ms) after which a grown KV soft cap is shrunk back to " + "--slot-initial-ctx and the over-cap pages are decommitted via posix_madvise " + "(default: %d, 0 disables; requires --slot-initial-ctx > 0)", + params.slot_shrink_idle_ms), + [](common_params & params, int value) { + params.slot_shrink_idle_ms = value; + } + ).set_env("LLAMA_ARG_SLOT_SHRINK_IDLE_MS").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_EMBEDDING, LLAMA_EXAMPLE_PARALLEL})); add_opt(common_arg( {"--cache-idle-slots"}, {"--no-cache-idle-slots"}, diff --git a/llama.cpp/common/common.cpp b/llama.cpp/common/common.cpp --- a/llama.cpp/common/common.cpp +++ b/llama.cpp/common/common.cpp @@ -1630,6 +1630,8 @@ struct llama_context_params common_context_params_to_llama(const common_params & cparams.n_seq_max = params.n_parallel; // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md cparams.slot_initial_ctx = params.slot_initial_ctx < 0 ? 0u : (uint32_t) params.slot_initial_ctx; + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + cparams.slot_shrink_idle_ms = params.slot_shrink_idle_ms < 0 ? 0u : (uint32_t) params.slot_shrink_idle_ms; cparams.n_rs_seq = params.speculative.need_n_rs_seq(); cparams.n_batch = params.n_batch; cparams.n_ubatch = params.n_ubatch; diff --git a/llama.cpp/common/common.h b/llama.cpp/common/common.h --- a/llama.cpp/common/common.h +++ b/llama.cpp/common/common.h @@ -430,6 +430,11 @@ struct common_params { // (Phase 1 behavior — only zero-fill is deferred). Positive values cap // the initial allocation and enable grow-on-demand up to n_ctx_seq. int32_t slot_initial_ctx = 0; + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Idle threshold (ms) after which a grown soft cap is shrunk back to + // slot_initial_ctx and the over-cap pages are decommitted via + // posix_madvise. 0 disables shrink-on-idle (Phase 2 behavior). + int32_t slot_shrink_idle_ms = 0; int32_t n_batch = 2048; // logical batch size for prompt processing (must be >=32 to use BLAS) int32_t n_ubatch = 512; // physical batch size for prompt processing (must be >=32 to use BLAS) int32_t n_keep = 0; // number of tokens to keep from initial prompt diff --git a/llama.cpp/include/llama.h b/llama.cpp/include/llama.h --- a/llama.cpp/include/llama.h +++ b/llama.cpp/include/llama.h @@ -346,6 +346,13 @@ extern "C" { // up to next_pow2 (still capped at n_ctx_seq). Reduces warm-state RSS // when typical prompts are much shorter than n_ctx_seq. uint32_t slot_initial_ctx; + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Idle threshold after which a grown soft cap is shrunk back to + // slot_initial_ctx and the pages beyond it are returned to the OS via + // posix_madvise(POSIX_MADV_DONTNEED). 0 disables shrink-on-idle + // (Phase 2 behavior — soft cap grows monotonically). Requires + // slot_initial_ctx > 0 to have any visible effect. + uint32_t slot_shrink_idle_ms; uint32_t n_rs_seq; // number of recurrent-state snapshots per seq for rollback (0 = no rollback) [EXPERIMENTAL] int32_t n_threads; // number of threads to use for generation int32_t n_threads_batch; // number of threads to use for batch processing diff --git a/llama.cpp/src/llama-context.cpp b/llama.cpp/src/llama-context.cpp --- a/llama.cpp/src/llama-context.cpp +++ b/llama.cpp/src/llama-context.cpp @@ -60,6 +60,8 @@ llama_context::llama_context( // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md cparams.slot_initial_ctx = params.slot_initial_ctx; + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + cparams.slot_shrink_idle_ms = params.slot_shrink_idle_ms; cparams.n_threads = params.n_threads; cparams.n_threads_batch = params.n_threads_batch; @@ -3350,6 +3352,7 @@ llama_context_params llama_context_default_params() { /*.n_ubatch =*/ 512, /*.n_seq_max =*/ 1, /*.slot_initial_ctx =*/ 0, + /*.slot_shrink_idle_ms =*/ 0, /*.n_rs_seq =*/ 0, /*.n_threads =*/ GGML_DEFAULT_N_THREADS, // TODO: better default /*.n_threads_batch =*/ GGML_DEFAULT_N_THREADS, diff --git a/llama.cpp/src/llama-cparams.h b/llama.cpp/src/llama-cparams.h --- a/llama.cpp/src/llama-cparams.h +++ b/llama.cpp/src/llama-cparams.h @@ -16,6 +16,11 @@ struct llama_cparams { // 0 = full n_ctx_seq allocation. Positive value caps the initial KV-cell // allocation per sequence and enables grow-on-demand up to n_ctx_seq. uint32_t slot_initial_ctx; + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // 0 disables shrink-on-idle. Positive value is the idle threshold in + // milliseconds: when a grown KV-cache is idle for at least this long, the + // soft cap is reset to slot_initial_ctx and the pages beyond are decommitted. + uint32_t slot_shrink_idle_ms; uint32_t n_rs_seq; // number of recurrent-state snapshots per seq for rollback int32_t n_threads; // number of threads to use for generation int32_t n_threads_batch; // number of threads to use for batch processing diff --git a/llama.cpp/src/llama-kv-cache-iswa.cpp b/llama.cpp/src/llama-kv-cache-iswa.cpp --- a/llama.cpp/src/llama-kv-cache-iswa.cpp +++ b/llama.cpp/src/llama-kv-cache-iswa.cpp @@ -21,6 +21,7 @@ llama_kv_cache_iswa::llama_kv_cache_iswa( bool unified, uint32_t kv_size, uint32_t kv_size_initial, + uint32_t slot_shrink_idle_ms, uint32_t n_seq_max, uint32_t n_ubatch, uint32_t n_pad, @@ -62,15 +63,15 @@ llama_kv_cache_iswa::llama_kv_cache_iswa( kv_base = std::make_unique( model, type_k, type_v, - v_trans, offload, unified, size_base, kv_size_initial, n_seq_max, n_pad, - 0, LLAMA_SWA_TYPE_NONE, filter_base, reuse); + v_trans, offload, unified, size_base, kv_size_initial, slot_shrink_idle_ms, + n_seq_max, n_pad, 0, LLAMA_SWA_TYPE_NONE, filter_base, reuse); LLAMA_LOG_INFO("%s: creating SWA KV cache, size = %u cells\n", __func__, size_swa); kv_swa = std::make_unique( model, type_k, type_v, - v_trans, offload, unified, size_swa, kv_size_initial, n_seq_max, n_pad, - hparams.n_swa, hparams.swa_type, filter_swa, reuse); + v_trans, offload, unified, size_swa, kv_size_initial, slot_shrink_idle_ms, + n_seq_max, n_pad, hparams.n_swa, hparams.swa_type, filter_swa, reuse); } void llama_kv_cache_iswa::clear(bool data) { diff --git a/llama.cpp/src/llama-kv-cache-iswa.h b/llama.cpp/src/llama-kv-cache-iswa.h --- a/llama.cpp/src/llama-kv-cache-iswa.h +++ b/llama.cpp/src/llama-kv-cache-iswa.h @@ -25,6 +25,9 @@ public: // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md // Forwarded to both kv_base and kv_swa. 0 = full kv_size. uint32_t kv_size_initial, + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Forwarded to both kv_base and kv_swa. 0 = shrink disabled. + uint32_t slot_shrink_idle_ms, uint32_t n_seq_max, uint32_t n_ubatch, uint32_t n_pad, diff --git a/llama.cpp/src/llama-kv-cache.cpp b/llama.cpp/src/llama-kv-cache.cpp --- a/llama.cpp/src/llama-kv-cache.cpp +++ b/llama.cpp/src/llama-kv-cache.cpp @@ -7,12 +7,35 @@ #include #include +#include #include #include #include #include #include +// opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md +// On Linux, madvise(addr, len, MADV_DONTNEED) is the call that actually +// decommits pages: the kernel zeroes the page table entries, releases the +// physical pages, and the next access faults in fresh zero-filled pages. +// glibc's posix_madvise(POSIX_MADV_DONTNEED) is documented as advisory and +// is NOT guaranteed to reduce RSS — empirically it is a no-op on the +// glibc/Debian-11 setup the embedder runs on. We must call madvise() +// directly. +// +// cosmocc's default headers (without _GNU_SOURCE) do not declare madvise() +// or MADV_DONTNEED, but the syscall exists on every Linux kernel the +// embedder runs on. Hardcode the Linux ABI value (asm-generic/mman-common.h: +// `#define MADV_DONTNEED 4`) and forward-declare madvise() to keep +// _GNU_SOURCE out of the global include. provides +// sysconf(_SC_PAGESIZE) for page-boundary math. +#include +#include +#ifndef MADV_DONTNEED +# define MADV_DONTNEED 4 +#endif +extern "C" int madvise(void *addr, size_t length, int advice); + static bool ggml_is_power_of_2(int n) { return (n & (n - 1)) == 0; } @@ -86,6 +109,7 @@ llama_kv_cache::llama_kv_cache( bool unified, uint32_t kv_size, uint32_t kv_size_initial, + uint32_t slot_shrink_idle_ms, uint32_t n_seq_max, uint32_t n_pad, uint32_t n_swa, @@ -115,6 +139,18 @@ llama_kv_cache::llama_kv_cache( __func__, kv_size_alloc, kv_size_max); } + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // The shrink target is the initial cap we just computed. last_active_ms_ + // starts at 0 (epoch) so a long pause before the first request still + // shrinks (correct behavior — nothing to lose). + kv_size_shrink_target = kv_size_alloc; + slot_shrink_idle_ms_ = slot_shrink_idle_ms; + last_active_ms_ = 0; + if (slot_shrink_idle_ms_ > 0 && kv_size_initial > 0 && kv_size_initial < kv_size_max) { + LLAMA_LOG_INFO("%s: slot_shrink_idle_ms = %u (target cap = %u)\n", + __func__, slot_shrink_idle_ms_, kv_size_shrink_target); + } + const uint32_t n_layer_kv = hparams.n_layer_kv(); // define a comparator for the buft -> ctx map to ensure that the order is well-defined: @@ -446,6 +482,151 @@ bool llama_kv_cache::ensure_capacity(uint32_t cells_needed) { return cells_needed <= kv_size_alloc; } +// opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md +// Decommit pages of every layer's K/V tensors in the byte-range that +// corresponds to cells [from_cells, to_cells). Returns those pages to the OS +// — RSS drops by the size of the range, and the next access to those pages +// will fault them back in as fresh zero-filled pages. +// +// Only safe to call after kv_size_alloc has been reset to from_cells (or +// lower) and n_cells_cleared has been reset accordingly, so the bookkeeping +// stays in sync with what's actually backed by physical memory. +// +// The decommit is per-stream because the KV tensors are 3-D +// [n_embd, kv_size, n_stream]; cells [a, b) within stream s occupy a +// contiguous byte-range whose start address is (base + s*nb[2] + a*nb[1]) +// and length is (b-a)*nb[1]. posix_madvise is page-grained — bytes that +// don't cover a full page are silently kept resident, which is fine for +// the cells that straddle page boundaries. +static void opencoti_decommit_layer_range(ggml_tensor * t, uint32_t n_stream, + uint32_t from_cells, uint32_t to_cells) { + if (!t || to_cells <= from_cells) { + return; + } + ggml_backend_buffer_t buf = t->buffer; + if (!buf) { + return; + } + // Only the host (CPU) backend's buffer is malloc'd memory we can + // decommit via madvise. For device buffers (CUDA/Metal/Vulkan) the K/V + // cache lives in device memory; skip — there are no host-side pages + // to return. This means Phase 3's RSS-shrink win applies to host-mode + // (e.g. embedder with -ngl 0); GPU-mode binaries still benefit from + // the soft-cap reset but RSS doesn't move. + if (!ggml_backend_buffer_is_host(buf)) { + return; + } + void * base = ggml_backend_buffer_get_base(buf); + if (!base) { + return; + } + const size_t stride_stream = (size_t) t->nb[2]; + const size_t stride_cell = (size_t) t->nb[1]; + const size_t off_in_buf = (size_t) ((char *) t->data - (char *) base); + for (uint32_t s = 0; s < n_stream; ++s) { + const size_t off = off_in_buf + + (size_t) s * stride_stream + + (size_t) from_cells * stride_cell; + const size_t sz = (size_t) (to_cells - from_cells) * stride_cell; + // posix_madvise wants the address — work in absolute terms. + void * addr = (char *) base + off; + // Round up to a page boundary on the low side, round down on the + // high side, so we only decommit pages we fully own. The cell-aligned + // address is unlikely to be page-aligned in general. + const size_t page = (size_t) sysconf(_SC_PAGESIZE); + if (page == 0 || page == (size_t) -1) { + return; + } + uintptr_t lo = (uintptr_t) addr; + uintptr_t hi = lo + sz; + uintptr_t lo_aligned = (lo + page - 1) & ~(uintptr_t)(page - 1); + uintptr_t hi_aligned = hi & ~(uintptr_t)(page - 1); + if (hi_aligned <= lo_aligned) { + continue; + } + // madvise(MADV_DONTNEED) returns 0 on success, -1 on failure (with + // errno set). We treat any failure as advisory — the RSS win may + // be skipped, but correctness is unaffected (the K/V buffer is + // still mapped, just possibly still resident). + (void) madvise((void *) lo_aligned, (size_t) (hi_aligned - lo_aligned), + MADV_DONTNEED); + } +} + +void llama_kv_cache::shrink_if_idle() { + if (slot_shrink_idle_ms_ == 0) { + return; // shrink-on-idle disabled + } + if (kv_size_alloc <= kv_size_shrink_target) { + return; // already at (or below) target — nothing to shrink + } + if (last_active_ms_ == 0) { + return; // no apply_ubatch has ever run — leave the initial cap as-is + } + + const uint64_t now = (uint64_t) + std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()).count(); + if (now < last_active_ms_) { + return; // clock went backwards; refuse to act + } + if ((now - last_active_ms_) < (uint64_t) slot_shrink_idle_ms_) { + return; // not idle long enough + } + + const uint32_t from_cells = kv_size_shrink_target; + const uint32_t to_cells = kv_size_alloc; + + // Evict prompt-cache cell metadata for indices [from_cells, to_cells) + // on every stream BEFORE decommitting the pages. The server's prompt + // cache (server-context.cpp) keeps cells populated between requests so + // a subsequent request with the same prefix can skip re-prefill. If we + // decommitted pages without invalidating cell metadata, the server's + // next prompt-cache lookup could "match" a prefix whose K/V data is + // now decommitted (returns to zero pages on next access) = corruption. + // + // Semantics: shrink-on-idle is allowed to evict the over-cap portion + // of the prompt cache because that is exactly what the user opted into + // by setting --slot-shrink-idle-ms > 0. A cache miss on a subsequent + // long prompt is the cost; the alternative (cap stays grown forever) + // defeats Phase 3 entirely — slots that retain cells between requests + // (the default for embed + completion) would never shrink. + uint32_t evicted = 0; + for (uint32_t st = 0; st < n_stream; ++st) { + auto & cells = v_cells[st]; + const uint32_t hi = std::min(cells.size(), to_cells); + for (uint32_t i = from_cells; i < hi; ++i) { + if (!cells.is_empty(i)) { + cells.rm(i); + ++evicted; + } + } + } + + LLAMA_LOG_INFO("%s: shrink soft cap %u -> %u (idle %llu ms, evicted %u cells)\n", + __func__, to_cells, from_cells, + (unsigned long long) (now - last_active_ms_), evicted); + + for (const auto & layer : layers) { + opencoti_decommit_layer_range(layer.k, n_stream, from_cells, to_cells); + opencoti_decommit_layer_range(layer.v, n_stream, from_cells, to_cells); + } + + // After decommit, treat cells [from_cells, to_cells) as "uncleared" — + // their pages will fault back in as fresh zero-filled pages on next + // touch, but n_cells_cleared still recorded them as cleared. Roll back + // the cleared mark so a future ensure_cleared() re-zeros them only when + // they're actually needed. + if (n_cells_cleared > from_cells) { + n_cells_cleared = from_cells; + } + kv_size_alloc = from_cells; + + // Refresh last_active_ms_ so we don't immediately try again on the very + // next init_batch (which would walk the empty layers for nothing). + last_active_ms_ = now; +} + void llama_kv_cache::clear(bool data) { for (uint32_t s = 0; s < n_stream; ++s) { v_cells[s].reset(); @@ -757,6 +938,12 @@ llama_memory_context_ptr llama_kv_cache::init_batch( bool embd_all) { GGML_UNUSED(embd_all); + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Check idle-shrink BEFORE ensure_cleared(): if shrink runs, it lowers + // kv_size_alloc and rolls n_cells_cleared back, so ensure_cleared's + // soft-cap-bounded clear targets only the (now smaller) initial cap. + shrink_if_idle(); + // opencoti F4 M3 Phase 1 — see docs/decisions/0001-lazy-slot-context.md ensure_cleared(); @@ -804,6 +991,12 @@ llama_memory_context_ptr llama_kv_cache::init_update(llama_context * lctx, bool } llama_kv_cache::slot_info_vec_t llama_kv_cache::prepare(const std::vector & ubatches) { + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Same reason ensure_cleared() is called here as well as init_batch: + // the iswa wrapper bypasses init_batch and calls prepare() directly on + // each inner kv_cache. shrink_if_idle is idempotent and cheap. + shrink_if_idle(); + // opencoti F4 M3 Phase 1 — see docs/decisions/0001-lazy-slot-context.md // The iswa wrapper calls prepare() directly on each inner kv_cache without // going through init_batch(), so the ensure_cleared() in init_batch alone @@ -1184,6 +1377,13 @@ llama_kv_cache::slot_info llama_kv_cache::find_slot(const llama_ubatch & ubatch, } void llama_kv_cache::apply_ubatch(const slot_info & sinfo, const llama_ubatch & ubatch) { + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Refresh the idle timestamp on every actual write. shrink_if_idle uses + // this to gate the shrink path. + last_active_ms_ = (uint64_t) + std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()).count(); + // keep track of the max sequence position that we would overwrite with this ubatch // for non-SWA cache, this would be always empty llama_seq_id seq_pos_max_rm[LLAMA_MAX_SEQ]; diff --git a/llama.cpp/src/llama-kv-cache.h b/llama.cpp/src/llama-kv-cache.h --- a/llama.cpp/src/llama-kv-cache.h +++ b/llama.cpp/src/llama-kv-cache.h @@ -106,6 +106,11 @@ public: // (Phase 1 behavior). Otherwise clamped to [n_pad, kv_size] // and rounded up to the next n_pad multiple. uint32_t kv_size_initial, + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Idle threshold (ms) after which the soft cap is shrunk + // back to kv_size_initial and the over-cap pages are + // decommitted. 0 disables shrink-on-idle. + uint32_t slot_shrink_idle_ms, uint32_t n_seq_max, uint32_t n_pad, uint32_t n_swa, @@ -180,6 +185,17 @@ public: // The current soft cap (cells visible to find_slot). uint32_t get_kv_size_alloc() const { return kv_size_alloc; } + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // If the soft cap has been grown above kv_size_shrink_target and at least + // slot_shrink_idle_ms_ have elapsed since the last apply_ubatch, reset + // the soft cap to kv_size_shrink_target and posix_madvise(POSIX_MADV_DONTNEED) + // the over-cap pages of every layer's K/V tensors. Idempotent and cheap + // when the soft cap is already at the target or shrink-on-idle is disabled. + // Called at the top of init_batch and prepare; safe because both happen + // before any slot work and the embedder's per-request cells.reset() means + // no slot ever holds onto cells across requests. + void shrink_if_idle(); + ggml_type type_k() const; ggml_type type_v() const; @@ -296,11 +312,21 @@ private: // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md // kv_size_max: hard cap on cells (set from constructor's kv_size). // kv_size_alloc: current soft cap. find_slot only scans cells [0, kv_size_alloc). - // Grows via ensure_capacity, never shrinks (Phase 2; Phase 3 - // adds shrink-on-idle). + // Grows via ensure_capacity. Phase 3 adds shrink-on-idle: + // shrink_if_idle() may reset it back to kv_size_shrink_target. uint32_t kv_size_max = 0; uint32_t kv_size_alloc = 0; + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Phase 3 shrink-on-idle state. kv_size_shrink_target is the cap value + // the soft cap is reset to on shrink — set once at ctor from + // kv_size_initial (or kv_size_max if Phase-1-mode). slot_shrink_idle_ms_ + // is the idle threshold in ms; 0 disables shrink. last_active_ms_ is a + // monotonic timestamp updated by apply_ubatch. + uint32_t kv_size_shrink_target = 0; + uint32_t slot_shrink_idle_ms_ = 0; + uint64_t last_active_ms_ = 0; + // the current index from where we start searching for a free slot in the ring buffer of KV cells (see find_slot()) // note: this is not part of the KV state and it's only used to speed-up the find_slot() method std::vector v_heads; diff --git a/llama.cpp/src/llama-memory-hybrid-iswa.cpp b/llama.cpp/src/llama-memory-hybrid-iswa.cpp --- a/llama.cpp/src/llama-memory-hybrid-iswa.cpp +++ b/llama.cpp/src/llama-memory-hybrid-iswa.cpp @@ -44,6 +44,10 @@ llama_memory_hybrid_iswa::llama_memory_hybrid_iswa( // Hybrid-iswa does not yet thread slot_initial_ctx; pass 0 for // Phase 1 backward-compat (full eager allocation). 0, + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Hybrid-iswa does not yet thread slot_shrink_idle_ms; pass 0 for + // Phase 2 backward-compat (no shrink-on-idle). + 0, n_seq_max, n_ubatch, n_pad, diff --git a/llama.cpp/src/llama-memory-hybrid.cpp b/llama.cpp/src/llama-memory-hybrid.cpp --- a/llama.cpp/src/llama-memory-hybrid.cpp +++ b/llama.cpp/src/llama-memory-hybrid.cpp @@ -43,6 +43,10 @@ llama_memory_hybrid::llama_memory_hybrid( // Hybrid memory does not yet thread slot_initial_ctx; pass 0 for // Phase 1 backward-compat (full eager allocation). 0, + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + // Hybrid memory does not yet thread slot_shrink_idle_ms; pass 0 for + // Phase 2 backward-compat (no shrink-on-idle). + 0, n_seq_max, n_pad, n_swa, diff --git a/llama.cpp/src/llama-model.cpp b/llama.cpp/src/llama-model.cpp --- a/llama.cpp/src/llama-model.cpp +++ b/llama.cpp/src/llama-model.cpp @@ -2075,6 +2075,8 @@ llama_memory_i * llama_model::create_memory(const llama_memory_params & params, cparams.n_ctx_seq, // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md cparams.slot_initial_ctx, + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + cparams.slot_shrink_idle_ms, cparams.n_seq_max, cparams.n_ubatch, 1, @@ -2093,6 +2095,8 @@ llama_memory_i * llama_model::create_memory(const llama_memory_params & params, cparams.n_ctx_seq, // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md cparams.slot_initial_ctx, + // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md + cparams.slot_shrink_idle_ms, cparams.n_seq_max, 1, hparams.n_swa,