docs/features/rys_probe.md

RYS template probe (--rys-probe) — design + implementation

Status: IMPLEMENTED (v1.1 = ΔPPL + graded task-probe → 2 templates + external eval hook), host-only, validated on Qwen3-4B (2026-07-13, #669/#670). Not yet captured as a patch / committed — ships on request. Companion to the shipped RYS execution engine, rys_layer_duplication.md (--repeat-layers, patch 0122).

0. What shipped (v1.1)

A new PROBE program mode in llamafile/rys_probe.cpp (dispatched from llamafile/main.cpp; flag detected in llamafile/args.cpp). It:

1. Parses args as the PERPLEXITY example, then disables -fit (the probe sizes its own small n_ctx; the fit path builds a trial context that we don't want mutating our cparams — it aborted the flash-attn print otherwise).
2. Loads the model once (common_init_from_params(params, /*model_only*/true)).
3. Enumerates contiguous mid-stack blocks [i,j) in the band (default [max(3,n/8) .. n-max(3,n/8))) × --rys-probe-widths. Default widths = auto = every width 1 .. (band) (an explicit W1,W2,… list overrides).
4. Scores base first (empty plan = reference), then each candidate by rebuilding only the context (llama_init_from_model with cp.repeat_layers = "i,j"; the model/weights are plan-independent), computing a compact self-contained ΔPPL over a small corpus (-f/-p, else a built-in slice) + a graded built-in task-probe (24 arithmetic / word-problem / factual / sequence items, greedy decode with substring-match early-stop; the fraction correct is the task score, used both as a rank signal and a guardrail — a candidate that drops below base task is discarded).
5. Prints a ranked table of the surviving wins, then exactly two ready-to-paste templates:
   • MOST EFFICIENT — best PPL-drop per added layer (largest ΔPPL/(#dup layers)).
   • MAX GAIN — largest absolute PPL-drop / lowest PPL. When no wider block beats the efficient one, the two coincide and the probe says so honestly ("the same block is both").

Flags: --rys-probe (enter probe mode) · --rys-probe-widths auto|W1,W2,… (default auto) · --rys-probe-band auto|i:j · --rys-probe-topk N (default 10, rows in the table). Corpus comes from the standard -f/-p; -ngl, -c, -fa flow through.

The last mile — external eval (c). ΔPPL + the built-in task-probe are cheap proxies, not a verdict. The probe's two templates feed perf/llamafile/rys-probe-eval.sh, which boots llama-server for BASE + each template (only --repeat-layers changes between arms) and runs a user-supplied EVAL_CMD (e.g. a GSM8K / MMLU slice via the omk harness) against each, printing a real task-score comparison with deltas-vs-base. That is the analogue of dnhkng's graded benchmark, minus the days-long search — the final pick is a downstream score, not a proxy.

Validation (Qwen3-4B-Q4_K_M, CPU):

• band [8,16), widths {2,4}, 12 candidates: base ppl 30.15; wins 9,13 (ppl→24.95, Δ−5.20, 11.1% over) and 10,12 (ppl→27.38, Δ−2.76, 5.6% over) → MAX GAIN = 9,13, MOST EFFICIENT = 10,12 (−1.38/layer), two distinct frontier points.
• band [10,16), widths auto (all 1..6), 14 candidates: base ppl 30.15 / task 0.917; winner 11,13 (ppl→28.03, Δ−2.12, 5.56% over, task 0.958) — sole survivor wide enough, so MOST EFFICIENT == MAX GAIN (honest "same block is both").

Contiguous mid-stack winners, no boundary layers, consistent with §1's structural prior. Build gotcha: a common_params struct field addition needs a clean host rebuild (rm -rf o + make) — incremental build:llamafile:make left ABI-skewed common_params consumers (bug-2160/2166 class); see BUILD_CYCLE.md. A later default-value change to an existing field (e.g. widths default 1,2,4→auto) is ABI-safe and rebuilds incrementally.

Question (user, 2026-07-13): can opencoti-llamafile grow a flag that probes a GGUF for good RYS layer-duplication templates, without reproducing dnhkng's multi-day search?

Short answer: yes, cheaply — because we can replace the two expensive halves of the reference method (a huge candidate space + a downstream-benchmark score) with (a) a pruned candidate space we already know the shape of, and (b) a perplexity scan the binary can already run. The result is a shortlist generator (minutes–~1 h), not a full Pareto search — the user still validates the top-K on the real retrieval/benchmark harness.

1. What the reference method (dnhkng RYS-II) actually does, and why it's slow

From https://dnhkng.github.io/posts/rys-ii/ (64-layer stack):

Stage	Mechanism	Cost
Full scan	every contiguous (i,j) block, duplicate [i,j), measure Δ	many single-block evals
Beam search	width 24, depth 3–6, greedily stack blocks	3,024 candidates, ~14 h
Surrogate	XGBoost trained on 4,411 measured configs ranks ~2,000,000 candidates	top 100 fully benchmarked
Validate	Pareto front (better and smaller) → 4 optimal configs	large val sets

Two things make it days-long:

1. The score is a downstream benchmark — Math120 (16–120 math Qs) + EQ140 (EQ-Bench scenarios). Every candidate needs generation + grading. Small "probe" sets are used for the search, larger sets for finalists, but it is still an accuracy eval per candidate.
2. The space is ~2 M contiguous+stacked blocks, so a surrogate (XGBoost on 4.4 k measured points) is needed just to rank it.

Key structural finding we can exploit: every Pareto-optimal template was a single contiguous mid-stack block. The cheapest winner was (33,34) (+1.56% params); the widest useful one (26,34) (+12.5%). Nothing near the first/last layers ever won — which is exactly our own boundary-layer result (rys_layer_duplication.md §10a, bug-2164: duplicating early/late layers yields incoherent output).

2. What opencoti already has

• The execution engine: --repeat-layers "i,j" sets cparams.layer_plan and re-runs [i,j) weight-shared, load-time, all archs (patch 0122). An in-process probe just calls this in a loop with different plans.
• A perplexity path in the vendored llama.cpp (used throughout the TCQ/KV campaigns) — a forward pass over a fixed corpus returning mean NLL.
• KLD-vs-reference tooling (logit-equiv harness).
• The boundary-layer advisory logic that already computes the safe mid-stack band [max(3,n/8) .. n-max(3,n/8)).

So the only missing piece is an outer loop + a scorer + a ranked report.

3. Proposed design — a PPL-scan shortlist, not a Pareto search

3a. Score: perplexity on a small fixed corpus (NOT KLD-to-base)

• Use ΔPPL vs the base model on a small held-out corpus. RYS improves the model by changing it, so a candidate that lowers PPL below base on a quality-correlated corpus is the cheap analogue of "Math/EQ went up". One forward pass over a few-k-token corpus per candidate — seconds to a couple of minutes on a served GPU, no generation/grading.
• KLD-to-base is the wrong optimization target here and we must not use it as the score: RYS intentionally diverges from base, so minimizing KLD rewards doing nothing (this is the same trap as "logit-equiv vs base" for RYS — rys_layer_duplication.md). KLD is still useful as a guardrail: a candidate whose KLD explodes (or whose PPL blows up) is the boundary-fragility / broken-merge regime — flag and drop it, don't rank it.
• Caveat, stated honestly: PPL is a weak proxy for reasoning/EQ gains. dnhkng used task accuracy precisely because self-merge gains are small and PPL-noisy. So the probe's job is to shortlist, and the last mile stays a real eval.

3b. Space: prune to contiguous mid-stack blocks (no surrogate needed)

Because winners are always contiguous mid-stack, enumerate only:

• contiguous blocks [i,j) fully inside the safe band [max(3,n/8) .. n-max(3,n/8)),
• with width 1 .. w_max (default w_max ≈ ceil(n/8) — the +12.5% ceiling),
• optionally strided starts to cap the count.

For a 64-layer model that is a few hundred candidates, not 2 M — so a direct scan needs no XGBoost surrogate at all. (An optional depth-2 greedy stack, à la their beam search, can be a follow-up once single-block ranking is trusted.)

3c. Surface: --rys-probe

llamafile --rys-probe \
  -m model.gguf \
  [--rys-probe-corpus calib.txt]   # default: bundled ~4k-token slice
  [--rys-probe-widths 1,2,4,8]     # block widths to try
  [--rys-probe-band auto|i:j]      # candidate band (default = advisory band)
  [--rys-probe-topk 10]            # how many to print / emit

Load the model once; for each candidate plan set layer_plan, run PPL over the corpus (base = empty plan, run first as the reference), record (block, overhead_pct, ppl, dppl_vs_base, kld_guardrail). Emit a table sorted by ΔPPL and a Pareto view (ΔPPL vs overhead%) — the same "better and smaller" lens dnhkng used, so the user picks a point on the frontier. Output the winner as a ready-to-paste --repeat-layers spec.

Reuses the 0122 engine + the existing perplexity path; host-only, no CUDA, no new kernels.

3d. Cost

N_candidates × one small-corpus PPL forward. A few hundred candidates × a few-k-token PPL ≈ minutes to ~1 h on one GPU for a mid-size model — vs days. The saving is entirely from (1) pruning 2 M → hundreds (structural prior) and (2) PPL instead of graded generation.

4. Honest limitations

• Shortlist, not verdict. PPL can rank a candidate above one that would win on reasoning/EQ. Ship it as "top-K to validate", and keep the real RULER/benchmark run as the decider — exactly how dnhkng re-measured finalists.
• PPL validity on franken-merges is model-dependent. For some Gemma-4 omnimerge GGUFs raw PPL is meaningless ([project note: PPL/KLD invalid on Gemma merges]); the probe must sanity-check the base PPL is finite/reasonable and warn (or fall back to a tiny task-probe) when it isn't.
• Single-block first. Stacked (beam) templates are a phase-2 extension; the single-contiguous-block scan already covers every dnhkng Pareto winner.

5. Recommendation

Feasible and cheap. Build it as an opt-in --rys-probe subcommand that (1) enumerates contiguous mid-stack blocks, (2) scores each by ΔPPL on a small fixed corpus with a KLD/PPL-blowup guardrail, (3) prints a ΔPPL-vs-overhead Pareto shortlist and the best --repeat-layers spec. It is a thin wrapper over the already-shipped 0122 engine + the existing perplexity path — no kernels, no CUDA. It does not reproduce dnhkng's rigor (no 2 M-candidate surrogate, no graded Math/EQ); it deliberately trades that for a minutes-scale shortlist the user finishes with one real eval pass.

Decision (user, 2026-07-13, #669/#670): shipped (b) ΔPPL + a graded arithmetic/short-reasoning task-probe, output narrowed to two templates (MOST EFFICIENT + MAX GAIN), plus a (c) external-eval hook (perf/llamafile/rys-probe-eval.sh) that runs a real downstream eval per template so the final pick is a task score, not a proxy. See §0.