# Kimi K3 GGML/RPC prefill upgrade plan (candidate-only) Date: 2026-08-01 This note is based on inspection of `/home/acer_gn100/llama.cpp-kimi-k3-candidate-20260801` and does **not** modify the live 8210 service or the 50052 workers. ## Highest-impact finding: PP is currently disabled The RPC device advertises these capabilities in `ggml/src/ggml-rpc/ggml-rpc.cpp`: ```cpp props->caps = { /* async = */ false, /* host_buffer = */ false, /* buffer_from_host_ptr = */ false, /* events = */ false }; ``` `src/llama-context.cpp` enables scheduler pipeline parallelism only when every non-CPU backend reports both `caps.async` and `caps.events`. Therefore the current layer-split RPC topology is a sequential split schedule, even though the normal llama scheduler calls `ggml_backend_sched_graph_compute_async`. Simply changing the two booleans is unsafe: the RPC backend has no event interface and the server currently computes each graph synchronously before it reads the next command. The candidate implementation must add a real fence. ## Upgrade 1 (largest): fenced asynchronous RPC + stage overlap Candidate protocol additions: ```cpp RPC_CMD_GRAPH_FENCE = ...; // request: device; response: completion status ``` On the worker, retain `graph_compute()` as the compatibility path, but add a candidate `graph_compute_async()` that queues `ggml_backend_graph_compute_async` and returns immediately. Add a per-device worker queue and a single completion thread (or CUDA event poller). `RPC_CMD_GRAPH_FENCE` calls `ggml_backend_synchronize(backends[device])` and returns only after all queued work is complete. The client implements `backend->synchronize` by sending the fence and implements event record/wait with monotonically increasing fence values. Only then set RPC `caps.async=true` and `caps.events=true`. Do not promote a boolean-only patch. It can overwrite stage inputs while a remote graph still reads them. Validation: run a two-ubatch synthetic graph with unique sentinels per ubatch; compare every stage output against sequential mode, then measure 2K/4K/8K prefill. Require zero RPC errors, no sentinel corruption, and no greedy output drift before enabling PP in the model launcher. ## Upgrade 2: pipelined RoCE transport (low protocol risk, candidate-only) `ggml/src/ggml-rpc/transport.cpp` currently uses one 256 KiB registered TX buffer and posts/polls one send completion for every chunk. This serializes large activation/expert copies. Candidate changes: * expose `GGML_RDMA_CHUNK` (default 256 KiB) and `GGML_RDMA_TX_DEPTH`; * allocate a TX ring (`TX_DEPTH * CHUNK`) and register it once; * post up to `TX_DEPTH` sends before polling completions; * poll the completion queue in batches; keep RX depth at least 24; * retain a 256 KiB fallback when memlock or QP limits reject the larger ring. Start with `CHUNK=1 MiB`, `TX_DEPTH=8` (8 MiB TX + existing 6 MiB RX fits the current memlock budget only if the budget is raised/confirmed). A conservative `CHUNK=512 KiB`, `TX_DEPTH=8` is the first candidate if memlock is unchanged. Measure a standalone 256 MiB RPC tensor copy and end-to-end prefill. Never claim a model speedup from the copy microbenchmark alone. ## Upgrade 3: avoid host round-trips for cross-stage tensors `ggml_backend_rpc_buffer_cpy_tensor()` only supports source and destination buffers on the **same** RPC socket. For different ranks the scheduler falls back to a blocking `get_tensor` into host RAM followed by `set_tensor`, creating two network transfers and a host copy at each stage boundary. Candidate protocol: `RPC_CMD_COPY_TENSOR_FROM_REMOTE` where the destination worker pulls from a source endpoint is not sufficient without RDMA connection management. Safer first step is an RPC peer-to-peer `COPY_TENSOR` command that uses a registered host-pinned staging ring and overlaps get/send/set with the next stage. Gate it behind `GGML_RPC_P2P_COPY=1`; fallback must remain exact. Validation: instrument bytes and wall time at every stage boundary; require the candidate to reduce host bytes and preserve output parity. This is likely more valuable for larger ubatches than for decode. ## Upgrade 4: use larger physical prefill microbatches Keep logical `--batch-size 2048` and candidate-A/B `--ubatch-size 768` then `1024`. Rank 3 is the limiting worker. Use the user's 1.5 GiB minimum memory floor, not a fictitious 12 GiB guard. Stop a candidate before allocation if the least-free rank would cross 1.5 GiB. Promote only if the fixed 2K/4K/8K natural prefill sweep improves and greedy output is unchanged. ## Upgrade 5: KDA chunked prefill / shared-gate kernels The 69-layer KDA path still loops over prompt tokens in one CUDA block. The default-off shared-gate patch removes repeated `expf(g)` evaluation but does not remove the recurrence. The real candidate is a state-carry chunk kernel (start with chunk 16 and 32), with serial reference fallback for ragged or unsupported layouts. Require parity at token lengths 1, 2, 31, 32, 33, 128, 512, and 2048, including terminal recurrent-state tolerance, before timing. ## Upgrade 6: IQ1_S SM121 tile tuning `mmq-config-blackwell.cuh` has native SM121 tiles for MXFP4/NVFP4 but IQ1_S falls through to Ampere MMQ tiles. Add an IQ1_S-only candidate tile table, benchmark the actual expert GEMM shapes, and retain only the best shape per `M/N/K` bucket. Do not globally force cuBLAS or MMQ; both can regress other ops. Gate on natural-output identity and per-kernel timings. ## Upgrade 7: graph reuse and cache/checkpoint policy Keep `--parallel 1`, exact-prefix caching, and `--cache-reuse 0`. For a truly append-only Hermes transcript, A/B `--ctx-checkpoints 0` to avoid large recurrent-state checkpoint copies. This is a memory/copy optimization, not a new-prompt prefill kernel, and must not be used when earlier messages are edited. `--no-cache-idle-slots` is defensive at one slot and is expected to be neutral. ## Candidate rollout order 1. Rebuild candidate with explicit `sm_121a` (already done); baseline parity. 2. Add shared-gate + KDA chunk kernels; numeric/state parity. 3. Run ubatch 768/1024 with 1.5 GiB floor. 4. Add RDMA TX ring; copy microbenchmark and fixed prefill sweep. 5. Add fenced RPC async only after correctness test; then enable scheduler PP. 6. Add cross-stage copy overlap and IQ1_S tile tuning. Every candidate run must record health, per-rank free memory, RPC errors, graph reuse, prompt tok/s, TTFT, decode tok/s, and greedy output hashes. Restore the known-good launcher automatically on any failed gate.