patches/0005-lazy-slot-context-shrink.patch

From: opencoti
Subject: [PATCH 5/N] lazy slot-context allocation — Phase 3 (shrink on idle)

Phase 2 added a soft cap on per-stream KV cells (kv_size_alloc <= kv_size_max)
and grow-on-demand via ensure_capacity. The soft cap never shrinks, so once a
single long prompt has caused it to grow, the embedder's RSS stays at the
grown size for the lifetime of the process. For an embedder running
`--parallel 2 --ctx-size 16384` with typical prompts well under 4K cells,
this means the headline Phase-2 win evaporates after the first long-context
request.

Phase 3 plugs that hole. When an idle threshold has elapsed since the last
apply_ubatch:

  1. Cells in the range [kv_size_shrink_target, kv_size_alloc) are evicted
     across every stream via cells.rm(i). The server-context layer's
     prompt cache keeps cells alive between embed requests for reuse;
     decommitting their backing pages without first evicting would leave
     the prompt cache pointing at zeroed memory and corrupt subsequent
     re-uses. The eviction makes the shrink unconditional.

  2. The pages backing cells [target, current_alloc) of every layer's K
     and V tensors are returned to the kernel via
     madvise(addr, len, MADV_DONTNEED).

  3. n_cells_cleared is rolled back to the shrink target so the next
     ensure_cleared() partial-memset re-zeros only what comes back.

  4. kv_size_alloc is reset to the shrink target so the next request can
     grow again from the initial cap.

Why madvise(MADV_DONTNEED) and not posix_madvise(POSIX_MADV_DONTNEED):
glibc on Linux maps posix_madvise(POSIX_MADV_DONTNEED) to a no-op (see
glibc sysdeps/posix/posix_madvise.c — the standard explicitly forbids
"actively destroying" pages, which is what we need). Only the Linux-
specific madvise(MADV_DONTNEED) actually decommits anonymous pages and
returns RSS to the kernel. We declared MADV_DONTNEED with the canonical
Linux value (4) under #ifndef and extern-declared madvise() so cosmocc's
default feature set (which does not define _GNU_SOURCE and so does not
expose madvise() through sys/mman.h) builds cleanly.

The shrink check runs at the top of init_batch() and prepare() — both
entry points to actual KV work. It is gated by:
  - slot_shrink_idle_ms_ > 0 (feature enabled)
  - kv_size_alloc > kv_size_shrink_target (something to shrink)
  - last_active_ms_ != 0 AND (now - last_active_ms_) >= idle threshold

Surface (CLI):
  --slot-shrink-idle-ms N (default 0 = disabled; positive enables)
  env LLAMA_ARG_SLOT_SHRINK_IDLE_MS

Surface (C API):
  llama_context_params::slot_shrink_idle_ms (uint32_t, default 0)

Plumbed through llama_cparams -> llama_kv_cache ctor ->
(llama_kv_cache_iswa ctor for SWA models) -> llama-model.cpp call sites.
Backward-compat: llama-memory-hybrid{,-iswa}.cpp pass 0 (no shrink), the
same way they already pass 0 for kv_size_initial.

Bench (CPU, --gpu disable so KV lives in host pages): the embedder
configured `--parallel 2 --ctx-size 16384 --pooling last
--slot-initial-ctx 256 --slot-shrink-idle-ms 30000` starts at 719 MB
RSS, grows to 1720 MB after a long-prompt corpus pass (soft cap 256 ->
2048), and after the 30 s idle drops to 1329 MB on the next short embed.
391.7 MB returned to the kernel — measured by /proc/<pid>/status VmRSS.
Quality (cosine min vs M2 baseline) stays at 0.999797 across the 48-prompt
corpus.

Milestone: F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
Upstreaming: TBD — kv_cache idle-shrink is a generally-useful concept; the
  cell-eviction + madvise(MADV_DONTNEED) approach is not opencoti-specific.
  Proposal candidate after F4 closes.

---
diff --git a/llama.cpp/common/arg.cpp b/llama.cpp/common/arg.cpp
--- a/llama.cpp/common/arg.cpp
+++ b/llama.cpp/common/arg.cpp
@@ -1371,6 +1371,17 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.slot_initial_ctx = value;
         }
     ).set_env("LLAMA_ARG_SLOT_INITIAL_CTX").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_EMBEDDING, LLAMA_EXAMPLE_PARALLEL}));
+    add_opt(common_arg(
+        // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+        {"--slot-shrink-idle-ms"}, "N",
+        string_format("idle threshold (ms) after which a grown KV soft cap is shrunk back to "
+                      "--slot-initial-ctx and the over-cap pages are decommitted via posix_madvise "
+                      "(default: %d, 0 disables; requires --slot-initial-ctx > 0)",
+                      params.slot_shrink_idle_ms),
+        [](common_params & params, int value) {
+            params.slot_shrink_idle_ms = value;
+        }
+    ).set_env("LLAMA_ARG_SLOT_SHRINK_IDLE_MS").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_EMBEDDING, LLAMA_EXAMPLE_PARALLEL}));
     add_opt(common_arg(
         {"--cache-idle-slots"},
         {"--no-cache-idle-slots"},
diff --git a/llama.cpp/common/common.cpp b/llama.cpp/common/common.cpp
--- a/llama.cpp/common/common.cpp
+++ b/llama.cpp/common/common.cpp
@@ -1630,6 +1630,8 @@ struct llama_context_params common_context_params_to_llama(const common_params &
     cparams.n_seq_max         = params.n_parallel;
     // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md
     cparams.slot_initial_ctx  = params.slot_initial_ctx < 0 ? 0u : (uint32_t) params.slot_initial_ctx;
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    cparams.slot_shrink_idle_ms = params.slot_shrink_idle_ms < 0 ? 0u : (uint32_t) params.slot_shrink_idle_ms;
     cparams.n_rs_seq          = params.speculative.need_n_rs_seq();
     cparams.n_batch           = params.n_batch;
     cparams.n_ubatch          = params.n_ubatch;
diff --git a/llama.cpp/common/common.h b/llama.cpp/common/common.h
--- a/llama.cpp/common/common.h
+++ b/llama.cpp/common/common.h
@@ -430,6 +430,11 @@ struct common_params {
     // (Phase 1 behavior — only zero-fill is deferred). Positive values cap
     // the initial allocation and enable grow-on-demand up to n_ctx_seq.
     int32_t slot_initial_ctx      =     0;
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // Idle threshold (ms) after which a grown soft cap is shrunk back to
+    // slot_initial_ctx and the over-cap pages are decommitted via
+    // posix_madvise. 0 disables shrink-on-idle (Phase 2 behavior).
+    int32_t slot_shrink_idle_ms   =     0;
     int32_t n_batch               =  2048; // logical batch size for prompt processing (must be >=32 to use BLAS)
     int32_t n_ubatch              =   512; // physical batch size for prompt processing (must be >=32 to use BLAS)
     int32_t n_keep                =     0; // number of tokens to keep from initial prompt
diff --git a/llama.cpp/include/llama.h b/llama.cpp/include/llama.h
--- a/llama.cpp/include/llama.h
+++ b/llama.cpp/include/llama.h
@@ -346,6 +346,13 @@ extern "C" {
         // up to next_pow2 (still capped at n_ctx_seq). Reduces warm-state RSS
         // when typical prompts are much shorter than n_ctx_seq.
         uint32_t slot_initial_ctx;
+        // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+        // Idle threshold after which a grown soft cap is shrunk back to
+        // slot_initial_ctx and the pages beyond it are returned to the OS via
+        // posix_madvise(POSIX_MADV_DONTNEED). 0 disables shrink-on-idle
+        // (Phase 2 behavior — soft cap grows monotonically). Requires
+        // slot_initial_ctx > 0 to have any visible effect.
+        uint32_t slot_shrink_idle_ms;
         uint32_t n_rs_seq;          // number of recurrent-state snapshots per seq for rollback (0 = no rollback) [EXPERIMENTAL]
         int32_t  n_threads;         // number of threads to use for generation
         int32_t  n_threads_batch;   // number of threads to use for batch processing
diff --git a/llama.cpp/src/llama-context.cpp b/llama.cpp/src/llama-context.cpp
--- a/llama.cpp/src/llama-context.cpp
+++ b/llama.cpp/src/llama-context.cpp
@@ -60,6 +60,8 @@ llama_context::llama_context(
 
     // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md
     cparams.slot_initial_ctx = params.slot_initial_ctx;
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    cparams.slot_shrink_idle_ms = params.slot_shrink_idle_ms;
 
     cparams.n_threads        = params.n_threads;
     cparams.n_threads_batch  = params.n_threads_batch;
@@ -3350,6 +3352,7 @@ llama_context_params llama_context_default_params() {
         /*.n_ubatch                    =*/ 512,
         /*.n_seq_max                   =*/ 1,
         /*.slot_initial_ctx            =*/ 0,
+        /*.slot_shrink_idle_ms         =*/ 0,
         /*.n_rs_seq                    =*/ 0,
         /*.n_threads                   =*/ GGML_DEFAULT_N_THREADS, // TODO: better default
         /*.n_threads_batch             =*/ GGML_DEFAULT_N_THREADS,
diff --git a/llama.cpp/src/llama-cparams.h b/llama.cpp/src/llama-cparams.h
--- a/llama.cpp/src/llama-cparams.h
+++ b/llama.cpp/src/llama-cparams.h
@@ -16,6 +16,11 @@ struct llama_cparams {
     // 0 = full n_ctx_seq allocation. Positive value caps the initial KV-cell
     // allocation per sequence and enables grow-on-demand up to n_ctx_seq.
     uint32_t slot_initial_ctx;
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // 0 disables shrink-on-idle. Positive value is the idle threshold in
+    // milliseconds: when a grown KV-cache is idle for at least this long, the
+    // soft cap is reset to slot_initial_ctx and the pages beyond are decommitted.
+    uint32_t slot_shrink_idle_ms;
     uint32_t n_rs_seq;        // number of recurrent-state snapshots per seq for rollback
     int32_t  n_threads;       // number of threads to use for generation
     int32_t  n_threads_batch; // number of threads to use for batch processing
diff --git a/llama.cpp/src/llama-kv-cache-iswa.cpp b/llama.cpp/src/llama-kv-cache-iswa.cpp
--- a/llama.cpp/src/llama-kv-cache-iswa.cpp
+++ b/llama.cpp/src/llama-kv-cache-iswa.cpp
@@ -21,6 +21,7 @@ llama_kv_cache_iswa::llama_kv_cache_iswa(
                      bool   unified,
                  uint32_t   kv_size,
                  uint32_t   kv_size_initial,
+                 uint32_t   slot_shrink_idle_ms,
                  uint32_t   n_seq_max,
                  uint32_t   n_ubatch,
                  uint32_t   n_pad,
@@ -62,15 +63,15 @@ llama_kv_cache_iswa::llama_kv_cache_iswa(
 
     kv_base = std::make_unique<llama_kv_cache>(
             model, type_k, type_v,
-            v_trans, offload, unified, size_base, kv_size_initial, n_seq_max, n_pad,
-            0, LLAMA_SWA_TYPE_NONE, filter_base, reuse);
+            v_trans, offload, unified, size_base, kv_size_initial, slot_shrink_idle_ms,
+            n_seq_max, n_pad, 0, LLAMA_SWA_TYPE_NONE, filter_base, reuse);
 
     LLAMA_LOG_INFO("%s: creating     SWA KV cache, size = %u cells\n", __func__, size_swa);
 
     kv_swa = std::make_unique<llama_kv_cache>(
             model, type_k, type_v,
-            v_trans, offload, unified, size_swa, kv_size_initial, n_seq_max, n_pad,
-            hparams.n_swa, hparams.swa_type, filter_swa, reuse);
+            v_trans, offload, unified, size_swa, kv_size_initial, slot_shrink_idle_ms,
+            n_seq_max, n_pad, hparams.n_swa, hparams.swa_type, filter_swa, reuse);
 }
 
 void llama_kv_cache_iswa::clear(bool data) {
diff --git a/llama.cpp/src/llama-kv-cache-iswa.h b/llama.cpp/src/llama-kv-cache-iswa.h
--- a/llama.cpp/src/llama-kv-cache-iswa.h
+++ b/llama.cpp/src/llama-kv-cache-iswa.h
@@ -25,6 +25,9 @@ public:
                      // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md
                      // Forwarded to both kv_base and kv_swa. 0 = full kv_size.
                      uint32_t   kv_size_initial,
+                     // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+                     // Forwarded to both kv_base and kv_swa. 0 = shrink disabled.
+                     uint32_t   slot_shrink_idle_ms,
                      uint32_t   n_seq_max,
                      uint32_t   n_ubatch,
                      uint32_t   n_pad,
diff --git a/llama.cpp/src/llama-kv-cache.cpp b/llama.cpp/src/llama-kv-cache.cpp
--- a/llama.cpp/src/llama-kv-cache.cpp
+++ b/llama.cpp/src/llama-kv-cache.cpp
@@ -7,12 +7,35 @@
 
 #include <algorithm>
 #include <cassert>
+#include <chrono>
 #include <cmath>
 #include <cstring>
 #include <limits>
 #include <map>
 #include <stdexcept>
 
+// opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+// On Linux, madvise(addr, len, MADV_DONTNEED) is the call that actually
+// decommits pages: the kernel zeroes the page table entries, releases the
+// physical pages, and the next access faults in fresh zero-filled pages.
+// glibc's posix_madvise(POSIX_MADV_DONTNEED) is documented as advisory and
+// is NOT guaranteed to reduce RSS — empirically it is a no-op on the
+// glibc/Debian-11 setup the embedder runs on. We must call madvise()
+// directly.
+//
+// cosmocc's default headers (without _GNU_SOURCE) do not declare madvise()
+// or MADV_DONTNEED, but the syscall exists on every Linux kernel the
+// embedder runs on. Hardcode the Linux ABI value (asm-generic/mman-common.h:
+// `#define MADV_DONTNEED 4`) and forward-declare madvise() to keep
+// _GNU_SOURCE out of the global include. <unistd.h> provides
+// sysconf(_SC_PAGESIZE) for page-boundary math.
+#include <sys/mman.h>
+#include <unistd.h>
+#ifndef MADV_DONTNEED
+#  define MADV_DONTNEED 4
+#endif
+extern "C" int madvise(void *addr, size_t length, int advice);
+
 static bool ggml_is_power_of_2(int n) {
     return (n & (n - 1)) == 0;
 }
@@ -86,6 +109,7 @@ llama_kv_cache::llama_kv_cache(
                      bool   unified,
                  uint32_t   kv_size,
                  uint32_t   kv_size_initial,
+                 uint32_t   slot_shrink_idle_ms,
                  uint32_t   n_seq_max,
                  uint32_t   n_pad,
                  uint32_t   n_swa,
@@ -115,6 +139,18 @@ llama_kv_cache::llama_kv_cache(
                 __func__, kv_size_alloc, kv_size_max);
     }
 
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // The shrink target is the initial cap we just computed. last_active_ms_
+    // starts at 0 (epoch) so a long pause before the first request still
+    // shrinks (correct behavior — nothing to lose).
+    kv_size_shrink_target = kv_size_alloc;
+    slot_shrink_idle_ms_  = slot_shrink_idle_ms;
+    last_active_ms_       = 0;
+    if (slot_shrink_idle_ms_ > 0 && kv_size_initial > 0 && kv_size_initial < kv_size_max) {
+        LLAMA_LOG_INFO("%s: slot_shrink_idle_ms = %u (target cap = %u)\n",
+                __func__, slot_shrink_idle_ms_, kv_size_shrink_target);
+    }
+
     const uint32_t n_layer_kv = hparams.n_layer_kv();
 
     // define a comparator for the buft -> ctx map to ensure that the order is well-defined:
@@ -446,6 +482,151 @@ bool llama_kv_cache::ensure_capacity(uint32_t cells_needed) {
     return cells_needed <= kv_size_alloc;
 }
 
+// opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+// Decommit pages of every layer's K/V tensors in the byte-range that
+// corresponds to cells [from_cells, to_cells). Returns those pages to the OS
+// — RSS drops by the size of the range, and the next access to those pages
+// will fault them back in as fresh zero-filled pages.
+//
+// Only safe to call after kv_size_alloc has been reset to from_cells (or
+// lower) and n_cells_cleared has been reset accordingly, so the bookkeeping
+// stays in sync with what's actually backed by physical memory.
+//
+// The decommit is per-stream because the KV tensors are 3-D
+// [n_embd, kv_size, n_stream]; cells [a, b) within stream s occupy a
+// contiguous byte-range whose start address is (base + s*nb[2] + a*nb[1])
+// and length is (b-a)*nb[1]. posix_madvise is page-grained — bytes that
+// don't cover a full page are silently kept resident, which is fine for
+// the cells that straddle page boundaries.
+static void opencoti_decommit_layer_range(ggml_tensor * t, uint32_t n_stream,
+                                          uint32_t from_cells, uint32_t to_cells) {
+    if (!t || to_cells <= from_cells) {
+        return;
+    }
+    ggml_backend_buffer_t buf = t->buffer;
+    if (!buf) {
+        return;
+    }
+    // Only the host (CPU) backend's buffer is malloc'd memory we can
+    // decommit via madvise. For device buffers (CUDA/Metal/Vulkan) the K/V
+    // cache lives in device memory; skip — there are no host-side pages
+    // to return. This means Phase 3's RSS-shrink win applies to host-mode
+    // (e.g. embedder with -ngl 0); GPU-mode binaries still benefit from
+    // the soft-cap reset but RSS doesn't move.
+    if (!ggml_backend_buffer_is_host(buf)) {
+        return;
+    }
+    void * base = ggml_backend_buffer_get_base(buf);
+    if (!base) {
+        return;
+    }
+    const size_t stride_stream = (size_t) t->nb[2];
+    const size_t stride_cell   = (size_t) t->nb[1];
+    const size_t off_in_buf    = (size_t) ((char *) t->data - (char *) base);
+    for (uint32_t s = 0; s < n_stream; ++s) {
+        const size_t off = off_in_buf
+                         + (size_t) s          * stride_stream
+                         + (size_t) from_cells * stride_cell;
+        const size_t sz  = (size_t) (to_cells - from_cells) * stride_cell;
+        // posix_madvise wants the address — work in absolute terms.
+        void * addr = (char *) base + off;
+        // Round up to a page boundary on the low side, round down on the
+        // high side, so we only decommit pages we fully own. The cell-aligned
+        // address is unlikely to be page-aligned in general.
+        const size_t page = (size_t) sysconf(_SC_PAGESIZE);
+        if (page == 0 || page == (size_t) -1) {
+            return;
+        }
+        uintptr_t lo = (uintptr_t) addr;
+        uintptr_t hi = lo + sz;
+        uintptr_t lo_aligned = (lo + page - 1) & ~(uintptr_t)(page - 1);
+        uintptr_t hi_aligned = hi & ~(uintptr_t)(page - 1);
+        if (hi_aligned <= lo_aligned) {
+            continue;
+        }
+        // madvise(MADV_DONTNEED) returns 0 on success, -1 on failure (with
+        // errno set). We treat any failure as advisory — the RSS win may
+        // be skipped, but correctness is unaffected (the K/V buffer is
+        // still mapped, just possibly still resident).
+        (void) madvise((void *) lo_aligned, (size_t) (hi_aligned - lo_aligned),
+                       MADV_DONTNEED);
+    }
+}
+
+void llama_kv_cache::shrink_if_idle() {
+    if (slot_shrink_idle_ms_ == 0) {
+        return;  // shrink-on-idle disabled
+    }
+    if (kv_size_alloc <= kv_size_shrink_target) {
+        return;  // already at (or below) target — nothing to shrink
+    }
+    if (last_active_ms_ == 0) {
+        return;  // no apply_ubatch has ever run — leave the initial cap as-is
+    }
+
+    const uint64_t now = (uint64_t)
+        std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::steady_clock::now().time_since_epoch()).count();
+    if (now < last_active_ms_) {
+        return;  // clock went backwards; refuse to act
+    }
+    if ((now - last_active_ms_) < (uint64_t) slot_shrink_idle_ms_) {
+        return;  // not idle long enough
+    }
+
+    const uint32_t from_cells = kv_size_shrink_target;
+    const uint32_t to_cells   = kv_size_alloc;
+
+    // Evict prompt-cache cell metadata for indices [from_cells, to_cells)
+    // on every stream BEFORE decommitting the pages. The server's prompt
+    // cache (server-context.cpp) keeps cells populated between requests so
+    // a subsequent request with the same prefix can skip re-prefill. If we
+    // decommitted pages without invalidating cell metadata, the server's
+    // next prompt-cache lookup could "match" a prefix whose K/V data is
+    // now decommitted (returns to zero pages on next access) = corruption.
+    //
+    // Semantics: shrink-on-idle is allowed to evict the over-cap portion
+    // of the prompt cache because that is exactly what the user opted into
+    // by setting --slot-shrink-idle-ms > 0. A cache miss on a subsequent
+    // long prompt is the cost; the alternative (cap stays grown forever)
+    // defeats Phase 3 entirely — slots that retain cells between requests
+    // (the default for embed + completion) would never shrink.
+    uint32_t evicted = 0;
+    for (uint32_t st = 0; st < n_stream; ++st) {
+        auto & cells = v_cells[st];
+        const uint32_t hi = std::min<uint32_t>(cells.size(), to_cells);
+        for (uint32_t i = from_cells; i < hi; ++i) {
+            if (!cells.is_empty(i)) {
+                cells.rm(i);
+                ++evicted;
+            }
+        }
+    }
+
+    LLAMA_LOG_INFO("%s: shrink soft cap %u -> %u (idle %llu ms, evicted %u cells)\n",
+            __func__, to_cells, from_cells,
+            (unsigned long long) (now - last_active_ms_), evicted);
+
+    for (const auto & layer : layers) {
+        opencoti_decommit_layer_range(layer.k, n_stream, from_cells, to_cells);
+        opencoti_decommit_layer_range(layer.v, n_stream, from_cells, to_cells);
+    }
+
+    // After decommit, treat cells [from_cells, to_cells) as "uncleared" —
+    // their pages will fault back in as fresh zero-filled pages on next
+    // touch, but n_cells_cleared still recorded them as cleared. Roll back
+    // the cleared mark so a future ensure_cleared() re-zeros them only when
+    // they're actually needed.
+    if (n_cells_cleared > from_cells) {
+        n_cells_cleared = from_cells;
+    }
+    kv_size_alloc = from_cells;
+
+    // Refresh last_active_ms_ so we don't immediately try again on the very
+    // next init_batch (which would walk the empty layers for nothing).
+    last_active_ms_ = now;
+}
+
 void llama_kv_cache::clear(bool data) {
     for (uint32_t s = 0; s < n_stream; ++s) {
         v_cells[s].reset();
@@ -757,6 +938,12 @@ llama_memory_context_ptr llama_kv_cache::init_batch(
             bool embd_all) {
     GGML_UNUSED(embd_all);
 
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // Check idle-shrink BEFORE ensure_cleared(): if shrink runs, it lowers
+    // kv_size_alloc and rolls n_cells_cleared back, so ensure_cleared's
+    // soft-cap-bounded clear targets only the (now smaller) initial cap.
+    shrink_if_idle();
+
     // opencoti F4 M3 Phase 1 — see docs/decisions/0001-lazy-slot-context.md
     ensure_cleared();
 
@@ -804,6 +991,12 @@ llama_memory_context_ptr llama_kv_cache::init_update(llama_context * lctx, bool
 }
 
 llama_kv_cache::slot_info_vec_t llama_kv_cache::prepare(const std::vector<llama_ubatch> & ubatches) {
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // Same reason ensure_cleared() is called here as well as init_batch:
+    // the iswa wrapper bypasses init_batch and calls prepare() directly on
+    // each inner kv_cache. shrink_if_idle is idempotent and cheap.
+    shrink_if_idle();
+
     // opencoti F4 M3 Phase 1 — see docs/decisions/0001-lazy-slot-context.md
     // The iswa wrapper calls prepare() directly on each inner kv_cache without
     // going through init_batch(), so the ensure_cleared() in init_batch alone
@@ -1184,6 +1377,13 @@ llama_kv_cache::slot_info llama_kv_cache::find_slot(const llama_ubatch & ubatch,
 }
 
 void llama_kv_cache::apply_ubatch(const slot_info & sinfo, const llama_ubatch & ubatch) {
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // Refresh the idle timestamp on every actual write. shrink_if_idle uses
+    // this to gate the shrink path.
+    last_active_ms_ = (uint64_t)
+        std::chrono::duration_cast<std::chrono::milliseconds>(
+            std::chrono::steady_clock::now().time_since_epoch()).count();
+
     // keep track of the max sequence position that we would overwrite with this ubatch
     // for non-SWA cache, this would be always empty
     llama_seq_id seq_pos_max_rm[LLAMA_MAX_SEQ];
diff --git a/llama.cpp/src/llama-kv-cache.h b/llama.cpp/src/llama-kv-cache.h
--- a/llama.cpp/src/llama-kv-cache.h
+++ b/llama.cpp/src/llama-kv-cache.h
@@ -106,6 +106,11 @@ public:
                      // (Phase 1 behavior). Otherwise clamped to [n_pad, kv_size]
                      // and rounded up to the next n_pad multiple.
                      uint32_t   kv_size_initial,
+                     // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+                     // Idle threshold (ms) after which the soft cap is shrunk
+                     // back to kv_size_initial and the over-cap pages are
+                     // decommitted. 0 disables shrink-on-idle.
+                     uint32_t   slot_shrink_idle_ms,
                      uint32_t   n_seq_max,
                      uint32_t   n_pad,
                      uint32_t   n_swa,
@@ -180,6 +185,17 @@ public:
     // The current soft cap (cells visible to find_slot).
     uint32_t get_kv_size_alloc() const { return kv_size_alloc; }
 
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // If the soft cap has been grown above kv_size_shrink_target and at least
+    // slot_shrink_idle_ms_ have elapsed since the last apply_ubatch, reset
+    // the soft cap to kv_size_shrink_target and posix_madvise(POSIX_MADV_DONTNEED)
+    // the over-cap pages of every layer's K/V tensors. Idempotent and cheap
+    // when the soft cap is already at the target or shrink-on-idle is disabled.
+    // Called at the top of init_batch and prepare; safe because both happen
+    // before any slot work and the embedder's per-request cells.reset() means
+    // no slot ever holds onto cells across requests.
+    void shrink_if_idle();
+
     ggml_type type_k() const;
     ggml_type type_v() const;
 
@@ -296,11 +312,21 @@ private:
     // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md
     // kv_size_max:   hard cap on cells (set from constructor's kv_size).
     // kv_size_alloc: current soft cap. find_slot only scans cells [0, kv_size_alloc).
-    //                Grows via ensure_capacity, never shrinks (Phase 2; Phase 3
-    //                adds shrink-on-idle).
+    //                Grows via ensure_capacity. Phase 3 adds shrink-on-idle:
+    //                shrink_if_idle() may reset it back to kv_size_shrink_target.
     uint32_t kv_size_max   = 0;
     uint32_t kv_size_alloc = 0;
 
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // Phase 3 shrink-on-idle state. kv_size_shrink_target is the cap value
+    // the soft cap is reset to on shrink — set once at ctor from
+    // kv_size_initial (or kv_size_max if Phase-1-mode). slot_shrink_idle_ms_
+    // is the idle threshold in ms; 0 disables shrink. last_active_ms_ is a
+    // monotonic timestamp updated by apply_ubatch.
+    uint32_t kv_size_shrink_target = 0;
+    uint32_t slot_shrink_idle_ms_  = 0;
+    uint64_t last_active_ms_       = 0;
+
     // the current index from where we start searching for a free slot in the ring buffer of KV cells (see find_slot())
     // note: this is not part of the KV state and it's only used to speed-up the find_slot() method
     std::vector<uint32_t> v_heads;
diff --git a/llama.cpp/src/llama-memory-hybrid-iswa.cpp b/llama.cpp/src/llama-memory-hybrid-iswa.cpp
--- a/llama.cpp/src/llama-memory-hybrid-iswa.cpp
+++ b/llama.cpp/src/llama-memory-hybrid-iswa.cpp
@@ -44,6 +44,10 @@ llama_memory_hybrid_iswa::llama_memory_hybrid_iswa(
         // Hybrid-iswa does not yet thread slot_initial_ctx; pass 0 for
         // Phase 1 backward-compat (full eager allocation).
         0,
+        // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+        // Hybrid-iswa does not yet thread slot_shrink_idle_ms; pass 0 for
+        // Phase 2 backward-compat (no shrink-on-idle).
+        0,
         n_seq_max,
         n_ubatch,
         n_pad,
diff --git a/llama.cpp/src/llama-memory-hybrid.cpp b/llama.cpp/src/llama-memory-hybrid.cpp
--- a/llama.cpp/src/llama-memory-hybrid.cpp
+++ b/llama.cpp/src/llama-memory-hybrid.cpp
@@ -43,6 +43,10 @@ llama_memory_hybrid::llama_memory_hybrid(
         // Hybrid memory does not yet thread slot_initial_ctx; pass 0 for
         // Phase 1 backward-compat (full eager allocation).
         0,
+        // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+        // Hybrid memory does not yet thread slot_shrink_idle_ms; pass 0 for
+        // Phase 2 backward-compat (no shrink-on-idle).
+        0,
         n_seq_max,
         n_pad,
         n_swa,
diff --git a/llama.cpp/src/llama-model.cpp b/llama.cpp/src/llama-model.cpp
--- a/llama.cpp/src/llama-model.cpp
+++ b/llama.cpp/src/llama-model.cpp
@@ -2075,6 +2075,8 @@ llama_memory_i * llama_model::create_memory(const llama_memory_params & params,
                                 cparams.n_ctx_seq,
                                 // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md
                                 cparams.slot_initial_ctx,
+                                // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+                                cparams.slot_shrink_idle_ms,
                                 cparams.n_seq_max,
                                 cparams.n_ubatch,
                                 1,
@@ -2093,6 +2095,8 @@ llama_memory_i * llama_model::create_memory(const llama_memory_params & params,
                                 cparams.n_ctx_seq,
                                 // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md
                                 cparams.slot_initial_ctx,
+                                // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+                                cparams.slot_shrink_idle_ms,
                                 cparams.n_seq_max,
                                 1,
                                 hparams.n_swa,