# RYS template probe (`--rys-probe`) — design + implementation

Status: **IMPLEMENTED (v1.1 = ΔPPL + graded task-probe → 2 templates + external
eval hook), host-only, validated on Qwen3-4B** (2026-07-13, #669/#670). Not yet
captured as a patch / committed — ships on request. Companion to the shipped RYS
execution engine,
[rys_layer_duplication.md](rys_layer_duplication.md) (`--repeat-layers`, patch 0122).

## 0. What shipped (v1.1)

A new `PROBE` program mode in `llamafile/rys_probe.cpp` (dispatched from
`llamafile/main.cpp`; flag detected in `llamafile/args.cpp`). It:

1. Parses args as the `PERPLEXITY` example, then **disables `-fit`** (the probe
   sizes its own small `n_ctx`; the fit path builds a trial context that we don't
   want mutating our cparams — it aborted the flash-attn print otherwise).
2. Loads the model **once** (`common_init_from_params(params, /*model_only*/true)`).
3. Enumerates contiguous mid-stack blocks `[i,j)` in the band (default
   `[max(3,n/8) .. n-max(3,n/8))`) × `--rys-probe-widths`. Default widths =
   **`auto`** = every width `1 .. (band)` (an explicit `W1,W2,…` list overrides).
4. Scores **base first** (empty plan = reference), then each candidate by
   rebuilding **only the context** (`llama_init_from_model` with
   `cp.repeat_layers = "i,j"`; the model/weights are plan-independent), computing
   a compact self-contained **ΔPPL** over a small corpus (`-f`/`-p`, else a
   built-in slice) **+ a graded built-in task-probe** (24 arithmetic / word-problem
   / factual / sequence items, greedy decode with substring-match early-stop; the
   fraction correct is the `task` score, used both as a rank signal and a guardrail
   — a candidate that drops below base `task` is discarded).
5. Prints a ranked table of the surviving wins, then **exactly two ready-to-paste
   templates**:
   - **MOST EFFICIENT** — best PPL-drop *per added layer* (largest `ΔPPL/(#dup layers)`).
   - **MAX GAIN** — largest absolute PPL-drop / lowest PPL.
   When no wider block beats the efficient one, the two coincide and the probe
   says so honestly ("the same block is both").

**Flags:** `--rys-probe` (enter probe mode) · `--rys-probe-widths auto|W1,W2,…`
(default `auto`) · `--rys-probe-band auto|i:j` · `--rys-probe-topk N` (default 10,
rows in the table). Corpus comes from the standard `-f`/`-p`; `-ngl`, `-c`, `-fa`
flow through.

**The last mile — external eval `(c)`.** ΔPPL + the built-in task-probe are cheap
*proxies*, not a verdict. The probe's two templates feed
[`perf/llamafile/rys-probe-eval.sh`](../../perf/llamafile/rys-probe-eval.sh),
which boots `llama-server` for BASE + each template (only `--repeat-layers`
changes between arms) and runs a user-supplied `EVAL_CMD` (e.g. a GSM8K / MMLU
slice via the omk harness) against each, printing a real task-score comparison
with deltas-vs-base. That is the analogue of dnhkng's graded benchmark, minus the
days-long search — the final pick is a downstream score, not a proxy.

**Validation (Qwen3-4B-Q4_K_M, CPU):**
- band [8,16), widths {2,4}, 12 candidates: base ppl 30.15; wins `9,13`
  (ppl→24.95, Δ−5.20, 11.1% over) and `10,12` (ppl→27.38, Δ−2.76, 5.6% over) →
  MAX GAIN = `9,13`, MOST EFFICIENT = `10,12` (−1.38/layer), two distinct frontier
  points.
- band [10,16), widths `auto` (all 1..6), 14 candidates: base ppl 30.15 / task
  0.917; winner `11,13` (ppl→28.03, Δ−2.12, 5.56% over, task 0.958) — sole survivor
  wide enough, so MOST EFFICIENT == MAX GAIN (honest "same block is both").

Contiguous mid-stack winners, no boundary layers, consistent with §1's structural
prior. Build gotcha: a `common_params` struct **field addition** needs a **clean
host rebuild** (`rm -rf o` + make) — incremental `build:llamafile:make` left
ABI-skewed `common_params` consumers (bug-2160/2166 class); see BUILD_CYCLE.md. A
later default-**value** change to an existing field (e.g. widths default
`1,2,4`→`auto`) is ABI-safe and rebuilds incrementally.

Question (user, 2026-07-13): can opencoti-llamafile grow a flag that *probes* a
GGUF for good RYS layer-duplication templates, without reproducing dnhkng's
multi-**day** search?

Short answer: **yes, cheaply** — because we can replace the two expensive halves
of the reference method (a huge candidate space + a downstream-benchmark score)
with (a) a **pruned** candidate space we already know the shape of, and (b) a
**perplexity scan** the binary can already run. The result is a *shortlist
generator* (minutes–~1 h), not a full Pareto search — the user still validates
the top-K on the real retrieval/benchmark harness.

## 1. What the reference method (dnhkng RYS-II) actually does, and why it's slow

From <https://dnhkng.github.io/posts/rys-ii/> (64-layer stack):

| Stage | Mechanism | Cost |
|---|---|---|
| Full scan | every contiguous `(i,j)` block, duplicate `[i,j)`, measure Δ | many single-block evals |
| Beam search | width 24, depth 3–6, greedily stack blocks | **3,024 candidates, ~14 h** |
| Surrogate | XGBoost trained on **4,411 measured** configs ranks **~2,000,000** candidates | top **100** fully benchmarked |
| Validate | Pareto front (better *and* smaller) → 4 optimal configs | large val sets |

Two things make it days-long:

1. **The score is a downstream benchmark** — `Math120` (16–120 math Qs) + `EQ140`
   (EQ-Bench scenarios). Every candidate needs generation + grading. Small
   "probe" sets are used for the search, larger sets for finalists, but it is
   still an accuracy eval per candidate.
2. **The space is ~2 M** contiguous+stacked blocks, so a surrogate (XGBoost on
   4.4 k measured points) is needed just to *rank* it.

**Key structural finding we can exploit:** every Pareto-optimal template was a
**single contiguous mid-stack block**. The cheapest winner was `(33,34)`
(+1.56% params); the widest useful one `(26,34)` (+12.5%). Nothing near the
first/last layers ever won — which is exactly our own boundary-layer result
([rys_layer_duplication.md §10a](rys_layer_duplication.md), bug-2164: duplicating
early/late layers yields incoherent output).

## 2. What opencoti already has

- The **execution engine**: `--repeat-layers "i,j"` sets `cparams.layer_plan` and
  re-runs `[i,j)` weight-shared, load-time, all archs (patch 0122). An
  in-process probe just calls this in a loop with different plans.
- A **perplexity** path in the vendored `llama.cpp` (used throughout the TCQ/KV
  campaigns) — a forward pass over a fixed corpus returning mean NLL.
- **KLD-vs-reference** tooling (logit-equiv harness).
- The **boundary-layer advisory** logic that already computes the safe mid-stack
  band `[max(3,n/8) .. n-max(3,n/8))`.

So the only missing piece is an *outer loop + a scorer + a ranked report*.

## 3. Proposed design — a PPL-scan shortlist, not a Pareto search

### 3a. Score: perplexity on a small fixed corpus (NOT KLD-to-base)

- **Use ΔPPL vs the base model on a small held-out corpus.** RYS improves the
  model by *changing* it, so a candidate that lowers PPL below base on a
  quality-correlated corpus is the cheap analogue of "Math/EQ went up". One
  forward pass over a few-k-token corpus per candidate — seconds to a couple of
  minutes on a served GPU, no generation/grading.
- **KLD-to-base is the wrong optimization target here** and we must not use it as
  the score: RYS *intentionally* diverges from base, so minimizing KLD rewards
  doing nothing (this is the same trap as "logit-equiv vs base" for RYS —
  [rys_layer_duplication.md](rys_layer_duplication.md)). KLD is still useful as a
  **guardrail**: a candidate whose KLD explodes (or whose PPL blows up) is the
  boundary-fragility / broken-merge regime — flag and drop it, don't rank it.
- Caveat, stated honestly: **PPL is a weak proxy for reasoning/EQ gains.** dnhkng
  used task accuracy precisely because self-merge gains are small and PPL-noisy.
  So the probe's job is to *shortlist*, and the last mile stays a real eval.

### 3b. Space: prune to contiguous mid-stack blocks (no surrogate needed)

Because winners are always contiguous mid-stack, enumerate only:

- contiguous blocks `[i,j)` fully inside the safe band `[max(3,n/8) .. n-max(3,n/8))`,
- with width `1 .. w_max` (default `w_max ≈ ceil(n/8)` — the +12.5% ceiling),
- optionally strided starts to cap the count.

For a 64-layer model that is a few hundred candidates, not 2 M — so a **direct
scan needs no XGBoost surrogate at all.** (An optional depth-2 greedy stack, à la
their beam search, can be a follow-up once single-block ranking is trusted.)

### 3c. Surface: `--rys-probe`

```
llamafile --rys-probe \
  -m model.gguf \
  [--rys-probe-corpus calib.txt]   # default: bundled ~4k-token slice
  [--rys-probe-widths 1,2,4,8]     # block widths to try
  [--rys-probe-band auto|i:j]      # candidate band (default = advisory band)
  [--rys-probe-topk 10]            # how many to print / emit
```

Load the model once; for each candidate plan set `layer_plan`, run PPL over the
corpus (base = empty plan, run first as the reference), record
`(block, overhead_pct, ppl, dppl_vs_base, kld_guardrail)`. Emit a table sorted by
ΔPPL and a **Pareto view** (ΔPPL vs overhead%) — the same "better *and* smaller"
lens dnhkng used, so the user picks a point on the frontier. Output the winner as
a ready-to-paste `--repeat-layers` spec.

Reuses the 0122 engine + the existing perplexity path; **host-only, no CUDA, no
new kernels.**

### 3d. Cost

`N_candidates × one small-corpus PPL forward`. A few hundred candidates × a
few-k-token PPL ≈ **minutes to ~1 h on one GPU** for a mid-size model — vs days.
The saving is entirely from (1) pruning 2 M → hundreds (structural prior) and
(2) PPL instead of graded generation.

## 4. Honest limitations

- **Shortlist, not verdict.** PPL can rank a candidate above one that would win
  on reasoning/EQ. Ship it as "top-K to validate", and keep the real
  RULER/benchmark run as the decider — exactly how dnhkng re-measured finalists.
- **PPL validity on franken-merges is model-dependent.** For some Gemma-4
  omnimerge GGUFs raw PPL is meaningless
  ([project note: PPL/KLD invalid on Gemma merges]); the probe must sanity-check
  the base PPL is finite/reasonable and warn (or fall back to a tiny task-probe)
  when it isn't.
- **Single-block first.** Stacked (beam) templates are a phase-2 extension; the
  single-contiguous-block scan already covers every dnhkng Pareto winner.

## 5. Recommendation

Feasible and cheap. Build it as an **opt-in `--rys-probe` subcommand** that
(1) enumerates contiguous mid-stack blocks, (2) scores each by ΔPPL on a small
fixed corpus with a KLD/PPL-blowup guardrail, (3) prints a ΔPPL-vs-overhead
Pareto shortlist and the best `--repeat-layers` spec. It is a thin wrapper over
the already-shipped 0122 engine + the existing perplexity path — no kernels, no
CUDA. It does **not** reproduce dnhkng's rigor (no 2 M-candidate surrogate, no
graded Math/EQ); it deliberately trades that for a minutes-scale shortlist the
user finishes with one real eval pass.

Decision (user, 2026-07-13, #669/#670): shipped **(b) ΔPPL + a graded
arithmetic/short-reasoning task-probe**, output narrowed to **two templates**
(MOST EFFICIENT + MAX GAIN), plus a **(c) external-eval hook**
(`perf/llamafile/rys-probe-eval.sh`) that runs a real downstream eval per template
so the final pick is a task score, not a proxy. See §0.