# Context tests vs needle-in-a-haystack These get mixed up. They are not the same measurement. | | question | what a fail means | |---|---|---| | Context tests | is the engine still computing the trained function at this length? | a bug | | Needle tests | can the model find a fact buried in a long prompt? | a 114M-model property | Fixing the engine will not make needle better. All numbers below are on the released `checkpoints/sft_7100.pt` (and `dpo_3200.pt` where noted). ## Context tests ```bash python engine_tools/check_context_ready.py --device cuda \ --ckpt checkpoints/sft_7100.pt --target-ctx 16384 ``` Most of the stack is position-local: hand it correct `position_ids` and length does not matter. Two pieces are not, which is why this script exists. - MemoryCacheBranch is segmented (`mc_segment_len 256`). Behaviour only changes when a sequence *crosses a segment boundary*. A short `verify.py` run (six-token prompt, twelve new tokens) never leaves segment 0, so it cannot see a cross-segment bug. - Engram carries an n-gram prefix across the cache boundary. The checks sit either side of L = 256 on purpose. | # | check | why | |---|---|---| | 1 | RoPE caches recompute from scratch exactly | `cos_cache` / `sin_cache` are `[max_positions, dim/2]` buffers saved in the weights. Widen the context and the loader drops them on a shape mismatch and builds new ones. This shows that drop is harmless, not just quiet. | | 2 | StaticKV footprint at the target context | 48 slots (loop × layer) is 3× a normal 16-layer model. Know the RAM before you allocate. | | 3 | Cache exactness across segment boundaries | the MC bug class. Prefill just under, just over, and two segments in. | | 4 | Cache exactness with decode crossing a boundary | the boundary happens mid-generation, not during prefill. Different path. | | 5 | Multi-turn window slide | conversation longer than context has to drop history, not overflow. | | 6 | `position_ids == max_position_embeddings` | must raise, not wrap to position 0. | 1 and 3–4 are the ones that actually catch a broken cache. ### 11/11 on `sft_7100` ``` rope cache recomputes from scratch exactly max |Δ| 0.000e+00 over 4096 positions (fractal=True) rope cache at 16384 agrees on first 4096 max |Δ| 0.000e+00 StaticKV, 48 slots × 16384 tok, bf16 0.40 GB prefill 252 tok (just under a segment) 6/6 identical, max |Δlogit| 3.05e-05 prefill 260 tok (just over) 6/6 identical, max |Δlogit| 3.82e-05 prefill 519 tok (two segments) 6/6 identical, max |Δlogit| 4.58e-05 decode crossing a boundary (prefill 253) 8/8 identical, max |Δlogit| 3.24e-05 window slide 3655 ≤ 4032 tok, history dropped position == ctx raises (index out of bounds) ``` `max |Δlogit| ≈ 3e-05` is the float32 floor for a 48-deep stack. Greedy is deterministic, so cached decode and a full recompute have to produce the same token ids. They do. That is "the cached path is the model", not "the cached path is close". ### Past the trained length Widen the config (`export_weights.py --seq-len 16384`) and the engine stays exact past 4,096: | prefill | tokens identical | max abs Δlogit | |---:|:---:|---:| | 4,096 | 6/6 | 3.34e-05 | | 6,000 | 6/6 | 2.96e-05 | | 8,192 | 6/6 | 4.20e-05 | This is engine correctness, not quality. Those positions got no gradient. It says a context extension is mechanically sound to try. It does not say the model will write anything useful out there without training the extension. ## Needle-in-a-haystack First pass put `The secret access code is BLUEBERRY-7291.` as sentence 1, padded with `The archive records routine maintenance notes for the facility.` to a target length, and asked `What is the secret access code?` at the end. Two scores: - keyword — did `BLUEBERRY` come back at all? - exact — did the full string, digits included, come back? Greedy was the primary arm (capability probe, no sampling noise). Sampled arm reported because that is what gets served. ### First pass — `sft_7100` (ctx 4,096) | prompt tokens | greedy: keyword | greedy: exact | temp 0.7: keyword | |---:|:---:|:---:|:---:| | 505 | yes | yes | yes | | 1,011 | yes | yes | yes | | 2,046 | yes | yes | no | | 3,587 | yes | yes | — | Greedy copied the digits out to ~3.6k. `dpo_3200` is exact at 505; its context is 1,024 so the longer rows do not apply. Temp 0.7 often mashed the suffix (`BLUEBERRY-dica`, `BLUEBERRY-Definition1`) and missed the keyword at 2k on that seed. Sampled column is one draw. That table is what was measured. It is not retrieval. Needle was the first sentence, filler was one sentence on a loop, code was a fixed string. Greedy wrote `The secret access code is BLUEBERRY-7291. The archive records r...` — the start of the prompt, copied. You pass this test by continuing from position 0. Nothing has to be looked up. ### Re-test (shortcut closed) Varied filler, a topic-specific question, a new randomly generated code every trial. | setting | exact | |---|--:| | fixed code, needle at depth 0, no distractors | 0.000 | | random code, depths 0/0.5/1.0, 3 distractors, 512–2048 tok | 0.056 | Typical miss: the frame is right, the digits are invented — `BLUEBERRY-?3`, `BLUEBERRY-Sch1`, `BLUEBERRY-tioxy`, `BLUEBERRY-ggregate`. It put out *a* code 33% of the time and it was the wrong one. Nothing code-shaped 61% of the time. It will echo a fixed, familiar string sitting at the top of the window. It does not reliably pull an arbitrary 4-digit code out of depth. This data does not split "can't retrieve" from "can't copy four random digits." Both fit. The window is not the only limit. The tests that actually changed the MC story are the paper's: S-NIAH-1/2 and MQAR on `base_62k.pt`, MC on vs `mc_gate` zeroed. Chance either way (S-NIAH-1 0/36 both; S-NIAH-2 1/36 on, 0/36 off; MQAR 0–1/20). That is in the card. This needle section is the old setup, left so nobody re-runs the continuation trick and calls it recall. A sibling-86M contrast from the first pass is dropped. Same broken setup, so it does not compare. ## Reproducing ```bash # context tests python engine_tools/check_context_ready.py --device cuda \ --ckpt checkpoints/sft_7100.pt --target-ctx 16384 # cache exactness alone (fast) python verify_cache.py --ckpt checkpoints/sft_7100.pt # first-pass needle (the continuation trick — not a retrieval test) python engine_chat.py --ckpt checkpoints/sft_7100.pt --temp 0 --max-new 30 \ -p " What is the secret access code?" ``` Both suites run on CPU (`--device cpu`) if the GPU is busy. Numbers match to the float32 floor, just slower.