# K3 PP/prefill candidate matrix (no live changes)

This is a bounded A/B plan for the 4× GB10 Kimi-K3 IQ1_S REAP568 run.  It keeps
the requested 1.5 GiB per-rank headroom (`1536 MiB`) and does not alter the
currently loading service.  Test one row at a time, with a 100-token decode
and a fixed cold/warm 2K/4K prompt pair.

## Current observed configuration

The active candidate is `sm_121a`, FlashKDA enabled, PP/layer split over RPC,
600K context, `batch=4096`, `ubatch=1536`, `threads=20`,
`threads-batch=32`, K=`q8_0`, V=`f16`, and three RPC workers on port 50053.
The head has 20 logical CPUs.  The current candidate build cache confirms
`GGML_RPC_RDMA=ON`, but NCCL is not found; llama.cpp therefore uses RPC rather
than NCCL for this path.

## Five high-value candidate upgrades

| ID | Change | Why it may help | Guard / rejection |
|---|---|---|---|
| U1 | **FlashKDA chunked prefill**: add a state-carry chunk path to `gated_delta_net.cu`, using 256–1024 token chunks and one FP32 terminal-state writeback. | The source has an explicit `TODO: Add chunked kernel for even faster pre-fill`; the current FlashKDA bridge only covers multi-token contiguous batches and the baseline KDA path remains serial. This removes the long serial scan from large PP microbatches. | Numeric parity against the existing N=1/N=2 bridge; reject on max output error >1e-3 or state error >1e-6. |
| U2 | **ubatch sweep**: test 1536 → 2048 → 3072 while holding logical batch 4096. | Larger physical microbatches amortize RPC/PP launch overhead and increase GEMM occupancy. The current 1536 setting is an unvalidated compromise, and ranks show >16 GiB available during load. | Keep rank-3 available ≥1536 MiB at peak; revert if cold TTFT or 2K/4K prefill worsens. |
| U3 | **CPU placement**: run `threads-batch=16–20` (not 32) and pin batch/tokenizer work to dedicated cores with `--cpu-range-batch`; keep RPC/control threads off that range. | Acer has 20 CPUs, so 32 batch threads oversubscribes the host and can compete with RPC. Earlier validated launch used 16/20. Pinning removes scheduler migration and protects the PP/RDMA progress path. | Compare CPU utilization and RPC p99; reject if worker link idle time rises or TTFT worsens. |
| U4 | **persistent RDMA path**: explicitly set the active RoCE device/GID and verify `RDMA activated` on all three connections; keep connections alive across requests. | Build is RDMA-enabled, but current worker logs only say “TCP (RDMA auto-negotiate enabled)”; they do not prove activation. The transport currently sends 256 KiB chunks with a 24-entry receive ring, which can underfill ConnectX. | Do not change the live service until logs prove RDMA activation. A/B with TCP fallback; reject on any RPC timeout or CRC/protocol error. |
| U5 | **RDMA transport batching**: compile a candidate with 1 MiB chunks and a 64-entry receive ring (memlock budget ≈64 MiB per connection), then benchmark PP prefill. | Current `RDMA_CHUNK=256 KiB` and `RDMA_RX_DEPTH=24` imply small-message/completion overhead. Larger chunks and deeper receives can improve long activation transfers. | Keep memlock bounded; reject if headroom or tail latency degrades. This requires a separate binary, never an in-place edit. |
| U6 | **graph-size ladder**: capture only the observed prefill sizes (256/512/1024/1536/2048) and decode 1/2/4/8, rather than a 600K graph. | Graph reuse only works when ubatch topology matches; the code explicitly compares token counts and sequence topology. A bounded ladder avoids graph rebuilds without allocating a monolithic 600K graph. | Capture incrementally; stop at 1.5 GiB floor. |
| U7 | **cache reuse and prefix discipline**: retain `--cache-reuse 256`, byte-identical system/tool prefix, and pre-tokenized prompt IDs. | The server supports cache reuse; it only helps repeated prefixes, not genuinely new prompts. This improves warm TTFT without changing model quality. | Report cold and warm separately; never claim cold-prefill gain from cache hits. |
| U8 | **PP boundary profiling**: record per-rank forward/RPC timings and move layer boundaries only if one rank is >10% slower. | `--split-mode layer` auto-balances memory, not necessarily KDA/GEMM execution time. KDA-heavy layers can create a stage bottleneck even when bytes are balanced. | Preserve the 1.5 GiB floor and verify all-rank output identity. |
| U9 | **Attention-K cache A/B**: test `--cache-type-k f16` (V remains f16) at 600K. | K q8 saves memory but adds quantize/dequant work in MLA attention. K3 has only 24 MLA layers; with `key_length=576`, the K f16 delta is roughly 2.1 GiB per node at 600K across four ranks, a plausible trade for faster prefill while retaining >1.5 GiB headroom. | Calculate actual allocator usage first; reject immediately if any rank approaches 1536 MiB or if long-context quality changes. |
| U10 | **IQ1_S SM121a tile candidate**: A/B the already-built isolated variant-1 binary (256-row/256-thread IQ1_S MMQ tile). | The stock IQ1_S path falls through Ampere MMQ tiles; the isolated candidate changes only IQ1_S dispatch geometry and is compiled for `sm_121a-real`. It can improve expert GEMM bandwidth without touching other quantizers. | Require greedy output identity and per-kernel timing improvement; never replace the current binary until a full worker parity check passes. |

## Suggested next non-destructive load

The next candidate should keep the already-built FlashKDA `sm_121a` binary and
try only U2 + U3 + U6 + U7:

```text
ctx=600000, parallel=1, batch=4096, ubatch=2048,
threads=16, threads-batch=20, cache-reuse=256,
flash-attn=on, split=layer, tensor-split=1,1,1,1,
K=q8_0, V=f16, FlashKDA=1, RPC=50053, guard=1536 MiB
```

U1/U4/U5/U8 are separate build or measurement work and should not be mixed
into the first A/B.  Success means faster **cold** 2K/4K prefill and no
regression in 100-token decode; warm results must be reported separately.