docs/evaluations/bug858-a4b-drafter-style-divergence.md

bug-858 A4B — drafter accept divergence on long_code_review (side-by-side)

Question: why does the gemma4-assistant drafter accept 0.90 of upstream (b9859) llama.cpp's tokens but only 0.26 of ours (llamafile) on this one prompt — while the two are at parity on the other 8 bench prompts?

Setup (identical on both sides)

• Target: google_gemma-4-26B-A4B-it-Q4_K_M.gguf (sha e718536f) — same file, both engines
• Drafter: gemma-4-26B-A4B-it-assistant.Q8_0.gguf (sha 5963a2ed) — same file
• Greedy (--temp 0), --parallel 1, ctx 4096, RTX 6000 Blackwell (bs2)
• Below: each engine's own no-spec thinking output for long_code_review, 640 tokens. (The drafter's job is to predict THIS stream token-by-token; that's what the accept-rate measures.)

Per-prompt accept (the gap is ONE prompt)

prompt	ours	b9859
code_python	0.87	0.92
code_cpp	0.90	0.90
explain_concept	0.77	0.81
summarize	0.92	0.92
qa_factual	0.83	0.92
translation	0.87	0.97
creative_short	0.785	0.7685
stepwise_math	0.91	0.89
long_code_review	0.2583	0.90
aggregate	0.7626	0.8866

Where it forks: a 0.01-logprob coin-flip on a newline (teacher-forced, pos 0-2)

pos 0:  ours '<|channel>'  == b9859 '<|channel>'    (identical, logprob 0.0)
pos 1:  ours 'thought'     == b9859 'thought'       (identical)
pos 2:  ours '\n'   logprob -0.69  (with '\n\n' 2nd at -0.70)   <- picks single newline
        b9859 '\n\n' logprob -0.67  (with '\n'  2nd at -0.71)   <- picks double newline

The two engines' logits agree to 0.01 logprob; they just land on opposite sides of a genuine ~50/50 tie. \n -> ours continues in prose; \n\n -> b9859 continues in bullets. That single newline is the entire cause of the style split below.

================================================================================ ## OURS (llamafile) — drafter accept 0.26

Python FastAPI, Postgres 15, 4 replicas, Nginx LB. Read user $\to$ Fetch 50 events (partitioned table) $\to$ Compute score $\to$ Update user $\to$ Insert history $\to$ Cache. p50 = 35ms, p99 = 1.8s (periodic, every ~90s). Elevated Postgres CPU during spikes. No correlation with connection count. 20 per replica (80 total). PgBouncer (transaction mode) pool size 50. users (50M), events (2B, partitioned by month, index on (user_id, ts DESC)), score_history (800M, index on user_id). Autovacuum at default. Nightly batch job (not the cause). PgBouncer waiting connections (8-12) during spikes. App CPU low. Network RTT low (0.4ms).

*   *The Transaction Block:* `async with db.transaction() as tx:` wraps everything.
    *   `SELECT ... FOR UPDATE`: This locks the user row.
    *   `SELECT ... FROM events`: This is a read.
    *   `UPDATE users`: This is a write.
    *   `INSERT INTO score_history`: This is a write.
*   *The Lock Duration:* The lock on the `users` row is held from the first `SELECT FOR UPDATE` until the end of the `async with` block.
*   *The "90-second" pattern:* This is a huge clue. Periodic spikes often suggest:
    *   Autovacuum/Maintenance tasks.
    *   Periodic background jobs.
    *   *Wait, what about the `score_history` table?* It's 800M rows.
    *   *What about the `events` table?* 2B rows, partitioned.
*   *The Postgres CPU spike:* High CPU without high connection count suggests heavy computation, index scans, or contention/locking overhead.
*   *The PgBouncer waiting connections:* This means the application is waiting for a connection from the pool. Since the app CPU is low, the app is blocked on I/O or waiting for a DB connection.

*   *Hypothesis 1: Lock Contention / Row Contention.*
    If multiple requests for the *same* `user_id` come in, they will queue on the `FOR UPDATE` lock. However, the prompt says "intermittent latency spikes" and "elevated Postgres CPU." Lock contention usually *lowers* CPU (processes are sleeping/waiting). High CPU suggests active work.

*   *Hypothesis 2: Autovacuum / Bloat / Index Maintenance.*
    The `score_history` table is 800M rows. Every request does an `INSERT`. This generates a lot of dead tuples

================================================================================ ## B9859 (upstream llama.cpp) — drafter accept 0.90

• Service: Python FastAPI (4 replicas).
• Database: Postgres 15.
• Load Balancer: Nginx.
• Connection Management: PgBouncer (Transaction mode, pool size 50).
• App Connection Pool: 20 per replica (Total 80).
• Workflow:
  1. SELECT ... FOR UPDATE on users (locks the row).
  2. SELECT from events (partitioned, 2B rows, index on (user_id, ts DESC)).
  3. Compute score (CPU in app).
  4. UPDATE users.
  5. INSERT INTO score_history (800M rows, index on user_id).
  6. cache.set (Redis/Memcached).
• Performance:
  • p50: 35ms.
  • p99: 1.8s (every 90s, regular pattern).
  • Correlation: High Postgres CPU, but not high connection count.
  • PgBouncer: Occasional waiting connections (8-12).
  • App CPU: Low (<30%).
  • Network RTT: Low (0.4ms).
• Schema:

  • users: 50M rows.

  • events: 2B rows, partitioned by month, index (user_id, ts DESC).

  • score_history: 800M rows, index user_id.

  • Autovacuum: Default.

  • Hypothesis 1: Lock Contention / Row Locking. The code uses SELECT ... FOR UPDATE. If the same user_id is hit frequently, requests will queue. However, the spikes are "regular" (every 90s) and affect Postgres CPU. Lock contention usually increases latency but doesn't necessarily spike CPU unless there's a massive amount of retries or context switching. But wait, if many requests are waiting for the same lock, they aren't consuming CPU; they are idle.

  • Hypothesis 2: Index/Table Bloat and Autovacuum. score_history is 800M rows and unpartitioned. Every request does an INSERT. This causes massive bloat. users is also updated. If autovacuum kicks in or if the index on score_history becomes extremely fragmented/large, it could cause spikes. But autovacuum is usually a background process. However, a "regular pattern" every 90 seconds suggests something periodic.

  • Hypothesis 3: Transaction Duration and Lock Holding. The transaction wraps the compute_score function. If compute_score is slow or if the `events

================================================================================

Verdict

Both continuations are coherent, competent code reviews (lock contention, autovacuum/bloat, the 90-second pattern — both get there). Ours is NOT degenerate. The only difference is FORMATTING: ours opens with a terse LaTeX-arrow summary ($\to$, $p50=35\text{ms}$) then settles into nested bullets; b9859 uses clean bullets from the first line. The distilled gemma4-assistant drafter predicts the standard bullet stream far better than ours' LaTeX-heavy prose stream — so it accepts 0.90 of b9859 and 0.26 of ours. The target itself is at parity (logits match upstream to 0.01 logprob); the accept gap is the drafter being style-sensitive, triggered by an unfixable floating-point newline coin-flip. Not a target bug, not degeneration.

================================================================================ ## FOLLOW-UP: is the LaTeX ours' degeneration? (teacher-forced, token-level)

Hypothesis (well-reasoned): nothing in the prompt cues LaTeX; nobody opens a reasoning trace with $\to$ / $\text{ms}$. If the drafter (distilled from true A4B) and upstream both avoid it, then ours emitting it = ours drifting off-distribution = degeneration.

Test: teacher-force ours' EXACT 24-token prefix into BOTH engines (/apply-template + /v1/completions logprobs) and read each engine's logprob for OURS' own next token. If ours is degenerate, upstream (the reference) will rank ours' LaTeX token low and greedily prefer a plain word/arrow. sane=Y means ours' greedy reproduced ours' own token (reconstruction valid).

pos | ours_next    |  o_lp | sane |   up_lp | up_greedy    | up_top3
  2 | '\n'         | -0.66 |   Y  |  -0.68  | '\n'         | ['\n' -0.7, '\n\n' -0.7, ')' -13.7]
  3 | 'Python'     | -0.21 |   Y  |  -0.20  | 'Python'     | ['Python' -0.2, 'Fast' -2.2, '*' -3.2]
  6 | ' Postgres'  | -0.09 |   Y  |  -0.06  | ' Postgres'  | [' Postgres' -0.1, ' ' -3.2, ' PostgreSQL' -4.4]
 15 | ' N'         | -0.19 |   Y  |  -0.24  | ' N'         | [' N' -0.2, ' nginx' -1.5, ' ' -8.6]
 20 | 'Read'       | -0.73 |   Y  |  -0.88  | 'Read'       | ['Read' -0.9, '`' -1.0, 'Reads' -2.4]
 21 | ' user'      | -0.04 |   Y  |  -0.03  | ' user'      | [' user' -0.0, ' `' -3.7, ' User' -4.8]
 22 | ' $\'        | -0.48 |   Y  |  -0.55  | ' $\'        | [' $\' -0.5, ',' -1.7, ' ->' -2.1]   <== the LaTeX

(full 0-23 in tf_ours.json / tf_up.json)

RESULT — hypothesis REFUTED. At the decisive LaTeX position (k=22), upstream's OWN greedy pick is $\ (the LaTeX), logprob -0.55 vs ours' -0.48 — and the plain-arrow -> is upstream's THIRD choice at -2.1, a full 1.6 logprob BELOW the LaTeX. Across all 23 positions ours' and upstream's logprobs for ours' tokens agree to ~0.1 and upstream's greedy == ours' token every time. Upstream avoids LaTeX from-scratch ONLY because the pos-2 newline coin-flip put it on the bullet path (numbered workflow), so the "Read user ->" prose context never arises. Put upstream ON that context and it writes the same LaTeX. => The LaTeX is a real A4B behavior both engines share on the prose path, not ours' degeneration. Ours' A4B decode is at genuine token-level parity with upstream.

================================================================================ ## CORRECTION (decisive): branch-forcing acceptance matrix — ours IS degraded

The teacher-forced test above validated the TARGET's output logits — but MTP acceptance depends on the DRAFTER (which eats the target's HIDDEN STATE), a path the logit test never touched. Forcing BOTH engines onto BOTH branches (identical prefix tokens, spec server, n1, temp 0) and measuring the drafter's acceptance:

            prose(\n)   bullet(\n\n)
ours          0.1286      0.4766
upstream      0.7753      0.8588

Read the COLUMNS. On the IDENTICAL bullet branch (same forced tokens; both emit near-identical * Service: Python FastAPI (4 replicas)...), ours=0.48 vs upstream=0.86 — a 0.38 gap that is NOT the branch. Upstream holds 0.78 even on the prose branch that collapses ours to 0.13. So:

• the branch / newline coin-flip is NOT the cause;
• ours' DRAFTER path is genuinely worse on EVERY branch of this prompt;
• the degeneration is REAL (in the drafter/acceptance path), not a benign tie-break.

The LaTeX turned out to be a red herring for the MECHANISM (upstream-forced-to-prose ALSO emits $\rightarrow$ — it IS a real A4B prose-path behavior) — but the underlying instinct "ours is degenerate" was CORRECT: it's just in the drafter path, not the target text.

Likely root cause: ours' hidden-state harvest / shared-KV drafter read is LENGTH-DEPENDENT wrong (8/9 short bench prompts are at parity; only long_code_review, ~760-token prefill, degrades). VERDICT REVERSED: #607 is NOT at parity; do NOT ship #608 as "parity achieved" — root-cause the ctx_dft drafter degradation at long prefill first.

---

RESOLUTION 2026-07-04 (identical-forced-prefix token comparison — SUPERSEDES the CORRECTION section)

The prior "CORRECTION" claimed ours' drafter is genuinely degraded because a branch-forcing matrix showed ours 0.48 vs upstream 0.86 on the "same" (forced-bullet) branch. That matrix was confounded: forcing only the 3-token prefix, then letting each engine generate freely, does NOT produce the same tokens. Decisive re-test (seqcmp-run.sh/seqcmp-up.sh, identical forced-bullet prefix bf_ids.json[bullet], temp 0, --parallel 1, n1, capturing return_tokens):

ours     accept 0.269   120 toks
upstream accept 0.815   120 toks
shared prefix = 6 tokens, then DIVERGE (ours=236761 "." vs up=568 " (")

Both continuations are COHERENT, non-degenerate bullet workflows — they simply diverge at token #6 from a benign per-token numerical coin-flip (ours' CUDA kernels vs upstream's), the same class as the earlier pos-2 \n/\n\n flip and #495 base-numerics:

• ours (accept 0.27): Service: Python FastAPI. · Database: PostgreSQL 15. · Deployment: 4 replicas, Nginx LB. · Workflow: 1. SELECT … FOR UPDATE on users · 2. SELECT … LIMIT 50 on events (partitioned) · 3. Compute score · 4. UPDATE users · 5. INSERT INTO score_history · 6. SET cache
• upstream (accept 0.82): Service: Python FastAPI (4 replicas). · Database: Postgres 15 (via PgBouncer in transaction mode). · Load Balancer: Nginx. · Workflow: 1. Start transaction. · 2. SELECT … FOR UPDATE on users · 3. SELECT … from events (partitioned, 2B rows, index on …) · 4. Compute score (CPU intensive? Not specified …

Ours' continuation is terser and more code/SQL-dense (backticks, table names, FOR UPDATE); upstream's is more discursive natural language. Code/SQL tokens are intrinsically higher-entropy = harder to self-draft than NL boilerplate — that is why ours' accept is lower on its own (different) content. The accept metric is measured over DIFFERENT token streams, so it never proved a drafter defect.

Verdict

• No degeneration. Ours is coherent on BOTH branches. The LaTeX seen from-scratch is a shared A4B behavior on the prose branch (upstream's own greedy picks it too — tf test), not ours-specific.
• Drafter is at parity. 8/9 bench prompts match upstream; the gemma4-assistant drafter runs mechanically correctly (DBG858 dual-ctx dump: posmax dft == tgt every step, real pending_h/ out_h, position tracks frontier; no truncation, no null-harvest).
• The residual aggregate gap is base-numerics content-divergence on knife-edge thinking prompts (ours' valid-but-denser continuation is harder to self-draft), NOT a ctx_dft port bug. Closing it would require bit-identical base kernels (#495 class) — out of scope for the MTP port.
• Auto-policies ruled out: sparse-V never armed (f16, not quantized-V), rolling-KV FULLY RESIDENT (inert), SWA irrelevant (n_swa=1024 ≫ 760-tok prefill).

Ship note: #607 dual-ctx port is functionally correct and drafter-at-parity → #608 may ship. Do NOT chase the 1/9 aggregate gap as a drafter fix — it is the #495 base-numerics rabbit hole. Evidence: /srv/ml/opencoti-c1/bug858-a4b-mtp/seqcmp_{ours,up}.json, dbg858b.server.log (DBG858).